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ABSTRACT 


The United States Naval Academy is the premier source of officers for the Naval service. 
It is a four-year total immersion educational experience designed to develop midshipmen 
morally, mentally, and physically and prepare them for service as either a Naval or Marine Corps 
officer. The Fourth-Class Indoctrination (Plebe Summer) program is the first military training 
evolution for most members of the incoming plebe class. The seven week indoctrination course 
is intended to provide an introduction to military and academy culture as well as to prepare 


fourth-class midshipmen for integration into the brigade of midshipmen. 


This research uses detailed literature reviews and expert interviews to identify the 
objectives and performance measurement procedures of Plebe Summer. Next it evaluates the 
performance measurement and outcome management procedures employed by the Naval 
Academy in assessing Plebe Summer success. Evaluated in this research are the objectives, 
outcomes, outcome indicators, data collection procedures, and benchmarks applied to the Plebe 
Summer program. The research indicates that a performance measurement and outcome 
management system can assist Naval Academy leaders in improving the quality of the Plebe 


Summer program. 
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I. INTRODUCTION 


A. BACKGROUND 


The United States Naval Academy (USNA) is the premier source of officers for 
the Naval Service. It provides a four-year totakimmersion program where a strong, 
balanced academic program, focusing on the education needs of the Naval Service is 
superimposed on a strict, professional military training environment (USNA, 1997). It is 
the mission of the Naval Academy to: 

Develop midshipmen morally, mentally, and physically and to imbue them 

with the highest ideals of duty, honor and loyalty in order to provide 

graduates who are dedicated to a career of naval service and have potential 


for future development in mind and character to assume the highest 
responsibilities of command, citizenship, and government. (USNA, 200Ic, 


p. 3) 
Throughout the more than 150- year history of the Naval Academy the institution 
has evolved from a trade school focused on the arts of seamanship to a highly regarded 


learning institution. The Naval Academy today provides academic majors in 19 areas 


including Engineering, Sciences, Humanities, and Economics. 


Fourth Class midshipmen (first-year freshmen) are introduced to the Naval 
Academy culture during an intense sevenweek course known as Fourth Class 
Indoctrination. This course is more commonly referred to as Plebe Summer and 
historically runs each year from late June through late August. The course is designed to 
be progressive and demanding training in subjects such as military drill, physical fitness, 
weapons, and naval history. These topics have been identified as those that are required 
to produce midshipmen who function effectively within the Academy and the Brigade of 


Midshipmen (USNA, 2001b, p.6). 
Plebe Summer is designed as a three phase course of instruction: 


1. First is the Processing Phase. This administrative phase is the shortest 
phase and is focused on insuring all midshipmen quickly make the administrative 


transition into the military culture. Areas of concern in this phase include official 
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welcomes, berthing assignment, uniform and equipment issue, medical evaluations, and 


academic testing (USNA, 2001b, p.8). 


Le Second is the Forming Phase. This phase is slightly more arduous 
mentally and physically. This phase is designed to instill a sense of teamwork and naval 
heritage in the midshipmen. Topics taught during this phase include, basic military 
indoctrination, close order drill, Navy rights and responsibilities. Activities emphasize 


discipline and teamwork (USNA, 2001b, p.8). 


3: The final phase is the Training Phase. This final phase is the longest and 
most challenging. The primary objective of this phase is progressive development 
through demanding training and follow-up to insure the incoming Plebe class meets the 
personal and professional standards expected of midshipmen at the Naval Academy 
(USNA, 2001b, p. 8). Major topics covered during this phase include moral and 
character development, code of conduct sailing and professional development, and 


military drill. 


Plebe Summer provides a basic foundation for further midshipman growth. The 
fundamental purpose of Plebe Summer is preparing midshipmen for the demands of the 
four-year Naval Academy challenge; it is a simply a first step. Upon completion of Plebe 
Summer midshipmen begin a rigorous year of training and education (called Fourth Class 
Development System) designed to challenge them in every aspect of their lives. The year 
is designed on the philosophy that, “to lead, one must first learn to follow.” (USNA 
2001e, p. 2) 


Performance measurement; and its extended application, outcome management, 
are tools gaining increasing popularity within both the private and public sectors. The 
core principle of outcome management is that program control is made more effective 
using detailed standards of performance that are measured and reported. “Increasingly 
companies are moving from management by opinion to management by fact — that is, 
away from soft science approach to performance measurement” (Harbour, 1997, p. 8). 
The ability of a high quality performance measurement program will assist program 


managers by providing relevant information quickly in order to support decision. “The 


goal of any performance measurement system is to provide the right people with the right 


performance-related information at the right time” (Harbour, 1997, p. 8). 


Persons accountable for programs apply performance measurement techniques for 
a number of reasons. Zigon (1998) lists three reasons for performance measurement and 


performance management that apply to the USNA: 
e You cannot manage what you cannot measure. 
e You cannot improve what you cannot measure. 
e High performance teams and individuals require clear goals. 
e Reward systems require metrics. 


There is clear evidence that performance measurement and outcome management 
are important aspects of managing the Plebe Summer program at the United States Naval 
Academy. Most fundamental is performance measurement’s ability to assist in 


evaluating the success of the Plebe Summer program in meeting its objectives. 


Performance metrics are usually linked with a program’s strategic plan. Strategic 
plans identify broad goals and objectives while performance metrics provide specific 
goals and measurable criteria that identify whether goals are met. Thus, performance 
objectives and metrics are intended to provide focus for those in management positions. 
“Leading organizations do not stop at the gathering and analysis of performance data; 
rather, these organizations use performance measurement to drive improvements and 
successfully translate strategy into action. In other words, they use performance 


measurement for managing their organizations” (PEA, 1998, p. vii). 


The Naval Academy developed its own detailed strategic plan in 1997. The plan 
is consistently updated (in 2001 and again in 2002) in order to maintain its relevancy. 
The plan was developed in response to governmental direction to implement mandatory 
performance measurement systems. All government agencies were directed by the 
Government Performance and Results Act of 1993 (GPRA) to develop comprehensive 
plans detailing the mission, goals, and objectives of the agency. Moreover, the GPRA 


requires a description of how these objectives are to be met and how performance goals 
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relate to the objectives of the agency. (United States Congress 1993) The USNA plan’s 
foundation lies in eight “focus areas”: leadership and professional excellence, physical 
fitness, naval heritage, academy quality of life, character building, effective 


communications, admissions excellence and academic excellence (USNA, 200Ic, p.8). 


Plebe Summer program serves as an indoctrination, and should be designed as an 
integral part of this strategic plan. The performance metrics should align with and 
support the goals of the plan. These goals will be addressed later in this thesis. 

B. PURPOSE 


The level at which the Naval Academy succeeds in meeting the Plebe Summer 
objectives is unknown. The objectives of Plebe Summer are diverse. In order to assure 
Fourth Class midshipmen succeed at the Naval Academy, they must be inculcated with 
Naval Academy and military cultures and be provided the tools and skills necessary to 
succeed within the culture during the seven-week Plebe Summer program. Furthermore, 
they must be prepared to function in demanding fields such as watch standing, teamwork, 
and military drill. Most importantly, upon completion of Plebe Summer, the midshipmen 
must be prepared to meet the challenges concerning their moral, mental, and physical 


development that will continue as the plebes join the brigade of midshipmen. 


The purpose of this study is to evaluate performance measurement and outcome 
management systems of the Naval Academy’s Pebe Summer Program and answer the 


following research questions: 
1. Primary: 


How effective are the performance measurement procedures used in the 
Plebe Summer program in evaluating the successful accomplishment of Plebe 


Summer objectives? 
2. Secondary: 
e What are the measurable objectives of Plebe Summer? 


e How is the success of Plebe Summer currently measured? 


e What current performance measurement procedures are used to assess 


the success of Plebe Summer objectives? 


e How will effective performance measurement techniques increase the 


effectiveness of the Plebe Summer program? 


Deeper understanding of the performance measurement and outcome management 
of Plebe Summer program can be beneficial to program managers. Answering these 
research questions will provide the following benefits to the Naval Academy and the 


Plebe Summer program: 


e Improving Plebe Summer effectiveness by identifying and _ separating 


procedures that enhance outcome management from procedures that do not. 


e Identifying objectives and outcomes of the Plebe Summer Program that are 


currently not assessed. 


e Improvements made in the basic foundation of midshipmen development will 
ultimately lead to higher levels of achievement of the Naval Academy’s 


overall objectives and mission. 


e Careful evaluation of the performance metrics applied to Plebe Summer may 
also illuminate areas of concern associated with incorrect applications of 
performance management. Additional motivation to close examination of 
Plebe Summer performance metrics is given by Perrin (1998) where he says 
close examination may identify flaws in performance management such as: 
varying interpretation of terms and concepts, meaningless measures, goal 
displacement where metrics become goals, lost focus on outcome, and 
misleading aggregate indicators (p. 5-10). 

Cc. METHODOLOGY 


This research is a qualitative assessment of the performance and outcome 
measurement processes used to assess the USNA’s Plebe Summer program in meeting its 
objectives. It includes literature reviews and personal semi-structured interviews of 


Naval Academy Plebe Summer experts. The research provides a qualitative assessment 


of outcome measurement methods using Hatry and Kopczynski’s Guide to Program 


Outcome Measurement (1997) as a standard. The process used in this research includes 


the following steps: 


A thorough literature review of previous performance measurement research. 
This review will include books, journal articles, internet, CD-ROM, and other 
electronic media materials dedicated to performance measurement and 


evaluation. 


A thorough review of all current USNA instructions and directives governing 
the management of the Plebe Summer program. This review will focus on the 
purpose, objectives, performance measurement and_ reporting, and 


accomplishment of Plebe Summer. 


Thorough semistructured interviews of expert Naval Academy personnel 
accountable for Plebe Summer planning, management, and evaluation. The 
purpose of these interviews is to establish and validate the performance 


metrics used in the Plebe Summer program. 


Evaluation of current USNA Plebe Summer performance metrics and outcome 
measurement processes using Hatry and Kopczynski’s Guide to Program 


Outcome Measurement (1997) as a standard of program evaluation. 


D. ORGANIZATION OF STUDY 


This study is organized in five chapters. The following briefly describes the 


contents each chapter of this thesis. 


Chapter I introduces the four basic aspects of this research: background, purpose, 


methodology, and organization of study. The background section describes the function 


of the United States Naval Academy and the Naval Academy’s mission, performance 


measurement as a method of management within both private and public sectors, and 


reviews the Naval Academy’s strategic plan focusing on its origins and purpose. 


Additionally, Chapter I describes the purpose of this research. It introduces the 


Plebe Summer program, as well as identifies the major research questions of this thesis. 


Lastly, this section proposes the benefits of successfully answering the research 


questions. 


Chapter II is divided into three areas. First, chapter two reviews previous 
performance measurement research and its application in the public sector. The review 
will focus on the purpose and objectives of performance measurement, terminology, 


current best practices, the misuse of performance measurement techniques. 


The literature review continues with a review of the current USNA strategic plan. 
The focus of this section is on the plan’s purpose and objectives. Lastly, chapter two 
reviews the current USNA Plebe Summer program. The examination focuses on the 


purpose, objectives, and methodology of the Plebe Summer program. 


Chapter III describes in detail Hatry and Kopczyski’s (1997) Guide to Program 
Outcome Measurement for the Department of Education This guide provides the model 
later used to analyze the Plebe Summer outcome and performance measurement 


processes. 


Chapter IV describes the qualitative research process of this thesis. This chapter 
summarizes the methodological philosophy used to develop the research methods. Also, 
this chapter outlines the scope and boundaries of the research, explains the interview 


selection process, and details the questions used in the semi-structured interview process. 


In chapter V Plebe Summer performance measurement and outcome management 
procedures identified in Chapter II and in expert interviews are evaluated using Hatry and 
Kopczyski’s (1997) Guide to Program Outcome Measurement. The results of this 
evaluation are in this chapter. Additionally, areas that are not currently covered by 


performance metrics are reported in this chapter. 


Chapter VI begins with a summary of the previous chapters. This last chapter 
continues with conclusions by making a qualitative decision answering the primary 
research question. Recommendations concerning Plebe Summer performance 
measurement as well as recommendations for further studies will follow the conclusions. 


Chapter VI will identify the connection between performance measurement, the strategic 


plan, and the management of the Plebe Summer program in an effort to better support the 


thesis’ answers of the research questions. 


Il. LITERATURE REVIEW 


A. INTRODUCTION 


This chapter begins with a review of current literature in the area of performance 
measurement and outcome management. Specific focus is given to the area of 
performance measurement. Next the chapter reviews the United States Naval Academy’s 
strategic plan and current performance measurement techniques. Finally, the chapter 
reviews the Naval Academy’s Plebe Summer program. The Plebe Summer review 
centers on the goals, objectives, procedures, and curriculum of the program as well as 
current performance and outcome measuring procedures. 


B. PERFORMANCE MEASUREMENT 


Leading and controlling organizations has grown nore difficult for a number of 
reasons. The size and complexity of organizations has increased. Additionally, the rate 
of technologic change has also accelerated. These changes coupled with increases in 
diversity among customers, employees, markets, and stakeholders create difficult 
management and control obstacles. Kaplan and Norton (1996) identify the necessity for 


business to change in the face of the “Information Revolution”: 


Companies are in the midst of a revolutionary transformation. Industrial 
age competition is shifting to information age competition. During the 
industrial age, from 1850 to about 1975, companies succeeded by how 
well they could capture the benefits from economies of scale to scope. 
Technology mattered, but, ultimately, success accrued to companies that 
could embed the new technology into physical assets that offered efficient, 
mass production of standard products (p. 2). 


However, they say, a new business environment has emerged and this new environment 


poses new challenges: 


The emergence of the information era, however, in the last decades of the 
twentieth century, made obsolete many of the fundamental assumptions of 
industrial age competition. No longer could companies gain sustainable 
competitive advantage by merely deploying new technology into physical 
assets rapidly, and by excellent management of financial assets and 
liabilities (Kaplan and Norton, 1996, p. 3). 
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New assumptions that provide the foundation of information age organizations are 
provided by Kaplan and Norton (996): integrated cross function business processes, 
integrated links to customers and suppliers, customized products to various customer 
segments, global scale in order to provide adequate returns, innovation, and investment in 
knowledge workers (p.4-5). Many organizations have recognized these new challenges 
and have attempted to improve their processes in order to meet them. However many 
organizations are not succeeding in meeting the new challenges. Kaplan and Norton 
(1996) offer performance management as a solution. 

Many of these improvement programs have yielded disappointing results. 

The programs are often fragmented. They may not be linked to the 

organization’s strategy, nor to achieving specific financial and economic 

outcomes. Breakthroughs in performance require major change, and that 
includes changes in the measurement and management systems used by an 
organization. Navigating to a more competitive, technological, and 


capability-driven future cannot be accomplished merely by monitoring and 
controlling financial measures of past performance (p.6). 


Attempting to overcome these obstacles, private sector organizations continually 
search for management systems that provide an economic edge over competitors. 
Performance measurement and management is one such system. World-class 
organizations use performance measurement systems to determine whether they are 
fulfilling their vision and meeting their customer-focused strategic goals (NPR, 1997, 
p.13). In short, performance measurement is the process of assessing progress toward 
organizational goals, including the efficiency with which resources are changed into 
outputs, the quality of outputs, outcomes achieved and the overall effectiveness of the 


organization’s efforts toward their mission (NPR, 1997, p. 4) 


In 1993, President Clinton signed into law the Government Performance and 
Results Act (GPRA). This legislation forced federal government agencies to adopt the 
practice of strategic management. The act lists six purposes for the GPRA, among those 


listed are: 


e Improve the confidence of the American people in the capability of the 
Federal Government, by systematically holding Federal agencies accountable 


for achieving program results. 
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e Improve Federal program effectiveness and public accountability by 


promoting a new focus on results, service quality, and customer satisfaction. 


e Help Federal managers improve service delivery, by requiring that they plan 
for meeting program objectives and by providing them with information about 


program results and service quality. 
e Improve internal management of the Federal Government (USC, 1993, p. 1). 


The GPRA requires all governmental agencies to design and submit strategic 
plans to the President and the Congress. Strategic plans are required to include agency 
goals, performance objectives, and performance metrics to indicate success or failure in 
meeting agency goals. Each governmental agency is then required to report, to the 
President and the Congress, annually on agency performance based on their strategic 
plans and performance objectives (USC, 1993, p. 1). Therefore, by signing the GPRA 
President Clinton institutionalized strategic management, performance management, and 


performance measurement throughout the federal government. 


Performance measurement and management has been widely studied and reported 
on. In general they have each received very positive support (Kaplan and Norton, 1996; 
Harbour, 1997; Arveson, 1998a; PEA, 1998; Zigon, 1998; NPR, 1999; Belz, 1999; Hatry, 
1997; Kamensky, 2001). Many public and private organizations have shifted (or are 
shifting) to management systems that include performance measurement techniques 
(PEA, 1998). These management shifts have been undertaken in order to better control 
various concerns such as productivity, efficiency, profitability, and customer and 


employee satisfaction (NPR 1997). 


However, performance measurement has not received universal praise (Scheirer, 
2000, Perrin, 1998; Arveson, 1998b, 1998c). Most of the concern about performance 
measurement and management centers on the fact that performance measures illustrate 
only how a company or program is performing. Performance measurement techniques 
cannot indicate why a company or program achieves its level of performance. The 
disconnect between performance measurement and causal forces continues to be a source 


of criticism throughout the literature (Schreier, 2000). 
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iF Goals And Objectives 


Performance management systems are designed to provide information required 
for the fundamental decisions and actions that shape and guide what an organization is, 
what it does, and why it does it. Performance measurement is a tool that when properly 
employed focuses organizational attention in order to better plan and execute strategic 
actions. The strategic focus provided by performance management is intended to result 
in improved organizational outputs and outcomes and is built on a foundation of 
performance measurement. NPR (1997) suggests that best-in-class organizations use 
measurement information to: set goals and standards; detect and correct problems; 


manage, describe, and improve processes; and document accomplishments (p.31). 


Performance measurement is one of many available tools that organizations use to 
manage their processes and control their organizational outputs and outcomes. The PEA 
(1998) describes how organizations exploit performance measurement and outcome 


management to assists in the management of their programs: 


One of the hallmarks of leading-edge organizations — be they public or 
private — has been the successful application of performance measurement 
to gain insight into, and make judgments about the organization and the 
effectiveness and efficiency of its programs, processes, and people. 
However, leading organizations do not stop at the gathering and analysis 
of performance data; rather these organizations use performance 
measurement to drive improvements and successfully translate strategy 
into action. In other words, they use performance measurement for 
managing their organizations (p. vii). 


The National Partnership for Reinventing Government (1997) echoes this view: 


All high-performance organizations are, and must be, interested in 
developing and deploying effective performance measurement and 
performance management systems, since it is only through such systems 
that they can remain high-performance organizations (p. 1). 


Additionally, Hatry and Kopczynski,(1997) say that not only performance should be 


measured. They express a need to also measure program outcomes: 


12 


Developing program outcome information is a critical step in developing a 
high quality performance measurement system for programs and projects 
for the federal government. Good program management requires 
collection and use of outcome data to provide guidance for improvement. 
Without information on results, managers can only supervise “inputs” or 
monitor processes. Decisions on whether the program is actually working 
well-or needs changing-are made in the absence of hard data on actual 
outcomes (p. 6). 


Organizations must first recognize, before taking action, the rationale on which 
performance measurement is founded. A great deal of the research postulates why 
organizations should undertake performance measurement (Averson, 1998a; Czarnecki, 
1999; Frost, 2000; NPR, 1997, 1999; PEA, 1998; Frost, 2000). There are four basic 


reasons most organizations undertake performance measurement: 


e You cannot manage what you cannot measure. In the absence of performance 
measurement managers cannot set goals, provide feedback, or communicate 


SUCCESS. 


e You cannot improve what you cannot measure. Without performance data 
managers cannot evaluate if program alterations result is actual program 


improvement. 


e High performance teams and individuals require clear goals. Clear goals are 
required so excellent performance can be recognized in highly diverse and 


intricate organizations. 


e Reward systems require metrics. When pay or rewards are based on 
performance, then managers require some way of knowing when reward has 


been earned (Zigon, 1998, p.1). 


Additionally, the National Center for Public Productivity (1997) identifies other benefits 
performance measurement provide public organizations such as greater accountability of 
managers, increase in service to public, and stimulation of public participation (NCPP, 


1997, p.3). 


The objective of performance management in the public sector can thus be 


defined as the detailed development and exploitation of performance metrics that 
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measure the success of organizational long-term and short-term strategic goals in order to 
increase service effectiveness. As performance management matures and becomes more 
established it becomes more important; thus it implies measure-based, strategy-driven 
organizational control (Frost, 2000). 


2. Performance Measurement Terminology 


Before getting deeper into performance measurement and _ performance 
management, it is useful to define the many terms used in this field. The literature offers 
a variety of similar but varying definitions for many of the performance measurement 
terms (NPR, 1997; Hatry and Kopczynski, 1997; Frost, 2000; Harbour, 1997). The 


following definitions will be used in this research to standardize the terms: 


Performance Measurement: The regular, ongoing measurement, and reporting 


on important performance aspects of organizational programs. 


Performance Measure: A quantitative or qualitative characterization of 


performance. 


Benchmark: What outcome indicators or performance measure information is 
compared to. Benchmarks provide a comparison for data. Benchmarks provide the 


minimum level of performance or change acceptable. 


Performance Management: The use of performance measurement information 
to help set agreed-upon performance goals, allocate and prioritize resources, inform 
managers to either confirm or change current organizational policies and procedures, and 


report the success in meeting organizational goals. 


Inputs: The amount of organizational inputs applied (i.e. amount of funds, time, 


or marhours). 


Outputs: Amount of organizational product completed (i.e. work processed, test 


scores, number and quality of products) 


Outcomes: Not what organizations do, but the consequences of their actions. 
Outcomes happen outside the organizations such as to customers or other organizations 


whose behavior the organization hopes to affect. Outcomes are classified as either 
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intermediate or end. Intermediate outcomes take actions that lead to end outcomes. End 


outcomes are the final desired results of the program’s work. 


Outcome Indicator: This is not the same as outcomes. Outcome indicators 
specify what is to be measured. Specific indicators will depend on the type of data 
collection used. Outcome indicators identify specific numerical values that indicate 


progress toward achieving an outcome (such as percentage or ratio). 


Outcome Measurement Process: The process for selecting outcome indicators 
and, subsequently, regularly obtaining and reporting data on the indicators. Outcome 


indicators are required for every significant program objective. 


Impact: The extent to which a program actually caused an outcome. 


3. Best Practices 


There are a variety of methods to employ the principles of performance 
measurement and performance management (Kaplan and Norton, 1996; Hatry and 
Kopczynski, 1997; NPR, 1997; PEA. 1998). Each approach possesses its own unique set 
of strengths and weaknesses, as well as challenges and opportunities. However, there are 
several characteristics of successful performance management systems that have been 
identified as likely to result in success. The National Partnership for Reinventing 
Government (1997) studied and reported on the performance management operations 
among 32 “best-in-class” organizations. A number of “best practices” were identified. 
Because world-class organizations use performance measurement systems to determine 
whether they are fulfilling their vision and meeting their customer-focused strategic 
goals, their performance measures must therefore meet the following criteria (NPR,1997, 


p.13): 


e Ensure narrow, strategic focus. The measures and goals an organization sets 
should be narrowly focused to a critical few. It is neither possible nor 
desirable to measure everything (NPR, 1997, p.13). Attempting to measure 
too much, or everything, leads to data overload. In this situation managers 


cannot distinguish important from irrelevant data. 
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e Measure the right thing. Before deciding on specific measures, an 
organization should identify and thoroughly understand the processes to be 
measured. Then, each key process should be mapped to ensure (1) a 
thorough, rather than assumed understanding of the process; and (2) that a 
measure central to the success of the process is chosen. In some cases, 
targets, minimums, or maximums are defined for each measure (NPR, 1997, 
p.13). This begins with first identifying what the organization must do well 
and developing a measure that represents this process (PEA, 1999, p.15). 


e Be a means, not an end. In a best-in-class organization, employees and 
managers understand and work toward the desired outcomes that are the core 
of their organization’s vision. They focus on achieving organizational goals 
by using performance measures to gauge goal achievement but do not focus 
on the measures per se. Performance measurement is thus seen as a means, 


not an end (NPR, 1997, p. 13). 


The NPR (1997), from their research, provides public sector agencies with key 
strategies to implement performance management systems that meet the GPRA 


requirements. The NPR recommends a severrstep process: 


1. Establish and update performance measures and goals. For each 
organizational goal or objective performance baseline, targets, and measures need to be 


established. In general a good measure: 
a. Is accepted by and meaningful to the customer; 
b. Tells how well goals and objectives are being met; 
c. Is simple, understandable, logical, and repeatable; 
d. Shows a trend; 
e. Is unambiguously defined; 
f. Allows for economical data collection; 


g. Is timely and sensitive; 
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The typical definition of measures includes specific goals, data requirements, calculation 


methodology, and rationale for the measure (NPR, 1997, p. 33-34). 


2. Establish accountability for performance. Both organizational and 
individual responsibilities need to be identified for the performance measures. In some 
organizations, goal targets do not become official until it is agreed to through a 
negotiation process between the goal owner and management. This ensures a high degree 


of integrity in the process and the people involved (NPR, 1997, p.34). 


3: Establish data collection process. Measures must be timely, easy to 
implement, and clearly defined. Speed is essential in data collection and distribution. 
Data collection should not be overly structured. Management should define who is 
responsible to ensure data collection is collected and processed accurately and reliably. 
Automation should be used whenever possible in order to reduce burden on the 


workforce (NPR, 1997, p. 35). 


4, Establish data analysis and reporting process. Various processes can be 
used to analyze, validate, and report measurement results. The most common processes 
are operations research, quality control, and statistical and qualitative analysis (NPR, 


1997, p. 36). 


D: Evaluate and use performance information. Performance information 
must be formally reviewed and ation upon to improve or simplify process. This 
provides management feedback for adjusting future performance plans and resources. 
The June 2, 1997 Federal Times cites several examples of agencies improving and 


refining performance measures: 


a. Instead of counting the number of forecasts it makes, the National 
Weather Service measures the warning time given to the public before severe 
weather. The lead-time before tornadoes increased from seven minutes to nine 


minutes. 


b. Instead of tracking the response time to questions about veterans’ 


eligibility for burial on its grounds, the National Cemetery System is trying to 
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improve the scheduling of burial services. Its goal is to finalize a date within two 


hours of receiving a request. (NPR, 1997, p. 36) 


6. Report performance to customers and stakeholders. Data should be 
reported and performance explained internally and externally across the organization. 
Reporting can be done using paper documents, internet, television, annual reports, or 


other multimedia methods (NPR, 1997, p.36). 


re Repeat the process. Sharing performance information with management, 
customers, and stakeholders facilitates deeper understanding among all parties. Each 
stakeholder can provide positive information to update and revise performance 
measurement systems for improvement. Therefore, the systems are not static and must 
constantly renew themselves (NPR, 1997, p. 37) 


4. Misuses of Performance Measurement 


Perrin (1998) provides valuable balance to the enthusiastic support performance 
measurement programs have received. His article provides insights to many of the 
limitations of performance measurement. He says, “Performance measurement 
programs, by themselves, are not appropriate for assessing outcomes, for determining 
future directions or for resource allocation. They can, however, be one important 
component of a comprehensive evaluation strategy” (p. 1). Moreover, he says that 
performance measurement programs are not living up to their billing as a means to ensure 
focus on results, produce greater accountability, or providing control over expenditures 


(Perrin, 1998, p. 2). 


Perrin (1998) identifies eight limitations of applying performance measures to 


determine program performance: 


1. Varying interpretations of the “same terms and concepts: Regardless of 
how clearly defined performance measures seem they can be interpreted differently 
among individuals. This inconsistency among staff can result in inaccuracies that make 


measures meaningless in the long term (Perrin, 1998, p. 4) 


Bi Goal Displacement: When measures become the objective, they result in 


“goal displacement”. This, in turn, emphasizes the wrong activities without improving 
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actual outcomes. The Viet Nam body count serves as a classic example of how too great 


a focus on measures reduced overall mission effectiveness (Perrin, 1998, p. 5). 


3: Use of meaningless or irrelevant measures: Performance measures are a 
simplification of a complex system. 

Attempting to reduce a complex program or social intervention, such as 

initiatives in child welfare, economic development, or health promotion, to 

a small number of quantitative indicators can disguise and mislead rather 

than inform what is really happening. Thus performance measures can 


confuse performance “indicators” with the underlying reality (Perrin, 
1998, p. 6). 


4, Cost savings vs. Cost shifting: Performance measures are unable to 
indicate underlying realities of a program. Therefore, they are unable to indicate if a 


performance measures are indicating true savings or simply shifting costs to the future. 


5. Critical subgroup differences disguised by misleading aggregate 
indicators: Performance measures often do not separate larger groups and apply 
performance data inaccurately across then group as a whole. This will tend to disguise 
indicators that could be more accurate if disaggregated by race, gender, age, education, 


etc (Perrin, 1998, p. 7) 


6. Limitations of objective-based approaches to evaluation: Performance 
management programs do not take a situational approach while applying performance 
data. Effective programs may be penalized by a performance management program for 
responding to changes in its environment that have not been updated in the overall 


performance management program (Perrin, 1998, p.8). 


Te Useless for decision making and resource allocation: Although one of the 
primary justifications for performance measurement is its ability to provide for more 
informed decision making, by itself, it is useless for this purpose. Performance measures 
can indicate what is happening in a program but it cannot indicate why. Therefore, 
managers must investigate further, and not rely on performance measures alone, when 


deciding on actions and resource allocations (Perrin, 1998, p.8) 
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8. Less focus on outcomes: Performance measurement leads to less rather 
than more focus on outcomes, innovation, and improvements. The results obtained from 
narrow focus on numbers limits staff empowerment, innovation, and self-evaluation. Too 
much focused placed on data collection leads to impaired performance and a reluctance 
to admit that improvement is required (Perrin, 1998, p. 8). 

C NAVAL ACADEMY STRATEGIC PLANNING AND PERFORMANCE 

MEASUREMENT 

1, Naval Academy Strategic Plan 


The United States Naval Academy (USNA) is the premier source of officers for 
the Naval Service. It provides a four-year totakimmersion program where a strong, 
balanced academic program, focusing on the education needs of the Naval Service is 
superimposed on a strict, professional military training environment (USNA, 1997, p.1). 
Throughout the more than 150-year history of the Naval Academy the institution has 
evolved from a trade school focused on the arts of seamanship to a highly regarded 
learning institution. The Naval Academy today provides academic majors in 19 areas 


including Engineering, Sciences, Humanities, and Economics. 


In response to the GPRA the USNA developed and published, in 1998, its first 
strategic plan to address its mission and vision. The strategic plan has been consistently 
updated yearly since its publication. The plan is designed to avoid mission drift, maintain 
appropriate balance between academic, professional, and athletic programs, and provides 
the foresight and focus to make decisions that will benefit the Brigade of Midshipmen 
and the Naval Academy for 10 years (USNA, 200Ic, p. 2). 


The USNA’s strategic plan has a pyramid structure as shown in Figure 1. The 
principal element of the plan is the Naval Academy’s vision. The vision of the strategic 
planners was to produce a Naval Academy that would: 

Provide leaders of great character, competence, vision and drive to 


transform the Navy and Marine Corps and serve the nation in a century of 
promise and uncertainty (USNA, 200Ic, p. 4) 


Supporting the vision is the Naval Academy’s mission: 
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Develop midshipmen morally, mentally, and physically and to imbue them 
with the highest ideals of duty, honor and loyalty in order to 








Focus Areas 


Program Initiatives 


Figure 1: Naval Academy Strategic Plan Structure [From: USNA, 200Ic. p. 3-6] 


provide graduates who are dedicated to a career of naval service and have 
potential for future development in mind and character to assume the 
highest responsibilities of command, citizenship, and government 
(USNA, 200Ic, p. 3). 


To support the vision and mission, the USNA strategic plan employs eight 


institutional focus areas. Each focus area is designed to address a different aspect of the 


Naval Academy and is described below. 


e Academic Excellence will focus on instructional and learning philosophies 
designed to promote critical thinking and lifelong learning. Moreover, this 
focus area is designed to provide faculty and staff resources to become and 


remain leaders in their respective disciplines. 


e Admissions Excellence will focus Academy efforts to attract the best qualified 


men and women willing to accept the challenges of the naval. 
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e Effective Communications is designed to provide Academy programs and 
support structures that will give midshipmen outstanding written and verbal 
communication skills. Also, it is designed to give midshipmen an opportunity 


to practice open, honest, and respectful communication skills. 


e Character Building focuses on developing midshipmen who are committed to 
selfless service, high moral standards, ethical behavior, and inculcated with 


the Naval Service’s core values of honor, courage, and commitment. 


e Leadership and Professional Excellence will prepare midshipmen for the 
challenges of fleet service in leadership roles. It will instill a deep respect for 


the Constitution and support the commitment to high moral standards. 


e Physical Fitness is intended provide midshipmen exemplary programs 
designed to physically challenge midshipmen and promote lifelong physical 


fitness. 


e Naval Heritage is a focus area intended to imbue midshipmen with an 
appreciation and respect of selfless service that are representative of naval 


heritage. 


e Academy Quality of Life receives focus to ensure a high quality of life for 
midshipmen and staff. This focus area is also intended to provide programs 
and facilities that enrich the cultural, recreational, and spiritual experiences for 


the various elements of the Naval Academy community (USNA, 200Ic, p. 9). 


The foundation of the USNA strategic plan is a constantly updated set of tactical 
and strategic initiatives. These initiatives are designed and implemented to improve 
current Academy facilities and programs or to create new programs where none 
previously existed. As the primary source of officers for the Navy and Marine Corps the 
Naval Academy has set very high quality goals for its graduates. The Academy strategic 


plan provides the following list of graduate attributes: 


e Prepared to lead in combat; 


an 


e Courageous leaders who take responsibility for their personal and professional 


decisions and actions; 
e Role models of ethical behavior and moral conduct; 
e Exemplars of academic, technical and tactical competence; 


e Individuals with a passion and commitment to lifelong learning and physical 


fitness; 
e Highly effective communicators; 


e Leaders who recognize and value individual excellence regardless of gender 


or cultural and ethnic background; 


e Able to understand and integrate geopolitical complexities in their decision 


making across the spectrum of military operations; 


e Patriots who epitomize the rich heritage, honor and traditions of the Navy, 
Marine Corps, and our country (USNA, 20014, p. 1) 


2. Naval Academy Performance Measurement 


The Naval Academy is currently using its strategic plan as the underpinning to 
plan and measure the development of midshipmen who exemplify the previous nine 
qualities. This section reports the current performance measurement used by the Naval 


Academy to monitor its progress and processes in achieving its vision and mission. 


Two theses were written in 1999 which investigated performance measurement 
techniques used at the Naval Academy (Belz, 1999; Boone, Hagen, and Utrotska, 1999). 
Both theses have two aspects in common. Both investigated the means by which Naval 
Academy Company Officers evaluated midshipmen development and both used Chang 


and DeYoung’s (1995) Measurement Linkage Model as the basis of their investigation. 


Each thesis identified the Naval Academy’s mission as the source of three key 
results areas: the moral, mental, and physical development of midshipmen. However, 
Boone et al. added an additional key results area: professional development. Belz and 


Boone et al. interviewed Company Officers to obtain key indicators for each key results 
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areas, analyzed the information obtained using Chang and DeYoung’s model, and then 
reported their findings. Figure 2 lists the key results areas and key indicators for each 


area listed by Belz and Boone et al. 


The theses provide insight into the use of performance measurement and 
management techniques used at the Naval Academy. The performance measurement 
process at the Naval Academy primarily focuses on three areas: 1) academic 
performance, 2) physical fitness, and 3) honor and conduct. Data is collected, stored, and 
retrieved using the Midshipman Information Data System (MIDS). The MIDS system, 
however, does not provide any analysis function. It is used only as a database and 


conduit of raw information. 


These two theses concentrate on the performance measurement procedures 
applied by Company Officers to assess performance on a company-wide scale. They do 
not represent performance measurement of Naval Academy programs. No research has 
been conducted focusing on all-encompassing performance management at the Naval 
Academy or performance measurement applied to individual programs or curriculum 


such as Plebe Summer. 


Surveys are an important performance measurement and outcome measurement 
tool (Hatry and Kopczski, 1997, p. 41). The Naval Academy annually conducts two 
important opinion surveys of midshipmen. First is the honor survey. This survey collects 
information pertaining to midshipmen perceptions of the honor concept and _ its 
applicability to the Academy. Second is midshipman summer-training survey. This 
survey collects midshipmen feedback concerning the usefulness and applicability of 
training received during summer sessions. Summer training includes professional 
practical education onboard ships, air squadrons, or United States Marine units. 

D. FOURTH CLASS INDOCTRINATION (PLEBE SUMMER) PROGRAM 


1. Introduction 


Fourth Class Indoctrination (Plebe Summer) program is the first phase in the 
Naval Academy’s year-long Fourth Class Midshipman Development System (USNA 
2001a, p. 1). The second phase of Fourth Class Development System is the Fourth Class 
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Academic Year that focuses on training, vice indoctrination which is the focus of Plebe 
Summer. The two phases together are designed to complement each other and ultimately 
lead to the transition of midshipmen from civilian and enlisted status to Naval Academy 
Midshipmen (USNA, 2001b, p. 5). This section reviews the purpose, goals, and 
objectives of the Plebe Summer program and details the program as outlined in Naval 


Academy Instructions. 


Belz 
Key Results Areas Key Indicators 


Moral Development None 


Mental Development GPA 
Absences 


Physical Development PE Grades 
Physical Readiness Test Scores 


Boone, Hagen & Utroska 

Key Results Areas Key Indicators 

Moral Development Honor Offenses/Counseling 
Conduct Offenses/Demerits 
Community Involvement 


Mental Development GPA 
Number of Honor Students 
Academic Extracurricular Activities 
Academic Boards/Probation/UNSATs 
Study Hours 


Physical Development Physical Readiness Test Scores 
PE Failures/Grades 
Weight and Body Fat Standards 


Professional Development Absences 
Professional Development Grades 
Uniform/Room Inspection Grades 
A/C Professional Quiz and Boards\ 
Performance Grades 





Figure 2: Belz and Boone et al. Key Results Areas and Key Indicators 


Fourth Class Indoctrination (Plebe Summer) serves as the introduction to the 


Naval Academy for midshipmen. The majority of midshipmen enter the Naval Academy 
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directly from high school and are experiencing their first extended period away from 
home. However, there are also a number of incoming midshipmen (approximately 20%) 
who enter the academy from fleet units or preparatory schools. Although these 
midshipmen possess experience in military culture, Plebe Summer serves as an 


introduction to the demands of the Naval Academy for them as well. 


Plebe Summer lasts for 49 days starting with Induction Day (-Day) and 
concluding with the reform of the brigade and the start of the academic year. Plebe 
Summer encompasses progressive and demanding training in those subjects required to 
produce midshipmen who function effectively in the brigade, are trained in drill, are 
physically fit, and practice those personal and professional traits which distinguish them 
as midshipmen at the United States Naval Academy (USNA, 2001b, p. 6). Plebe training 
is not designed as an “initiation”. It is, rather, an indoctrination to military and academy 
life intended to prepare fourth-class midshipmen for the rigorous challenges they will 


face during the four-year academy curriculum. 
The Naval Academy lists the following Plebe Summer objectives: 
e Effect the transition from civilian or enlisted to officer candidate status. 


e Indoctrinate and train midshipmen in proper military discipline, courtesy, 
requirements and traits of special trust and confidence, teamwork, and 


personal accountability. 


e Provide basic training in seamanship, marksmanship, military drill, watch 
standing, and customs and traditions of the Naval service (USNA, 2001b, p. 
6). 


There are three distinct phases of Plebe Summer: the Processing, Forming, and 
Training phases. As midshipmen progress from one phase to another they face 
increasingly stressful and challenging environments. The following reports the purpose 


and primary focus of each phase. 


The Processing Phase is designed to facilitate the administrative transition from 


civilian to military life. This includes equipment and uniform issue, room set-up, and all 


26 


the necessary administrative transition paperwork. As much as feasible, the schedule is 
filled with military instruction in a low stress environment (USNA, 2001b, p. 8). The 
Processing Phase lasts for four days. Subjects focused on during this phase include 
academic testing, introduction to religious programs, medical physicals, swim testing, 


and marksmanship testing. 


Training-day five marks the beginning of the Forming Phase. This phase is 
distinguished from the Processing Phase by an increase in stress and pace of activities. 
The purpose of this segment is to concentrate on basic military indoctrination, instilling 
discipline, developing teamwork, and promoting a good sense of Naval Heritage (USNA, 
2001b, p. 8). This phase is designed to onto the midshipmen a foundation of base- level 
knowledge and spirit that will be used in the last phase of Plebe Summer. Important 
subjects covered during this period include close-order drill, character development 


education, introduction to Navy rights and responsibilities, and professional counseling. 


The last and longest phase is the Training Phase. The primary objective of this 
phase is continued progressive development through demanding training and follow-up to 
ensure the incoming Plebe class meets the personal and professional standards expected 
of midshipmen (USNA, 2001b, p. 8). Upon completion of the Training Phase 


midshipmen are expected to meet the following training objectives: 
e Know all basic rates (professional knowledge requirements) 
e Properly wear all uniforms 
e Be able to provide directions to any significant location on campus 
e Be qualified in small sailing craft and small arms 


e Be qualified to properly stand military watches to include: Company Mate of 
the Deck (CMOD), and Main Office Messenger (MOM) 


e Demonstrate a complete working knowledge of Midshipman Regulations 


e Achieve at least a grade of “C” on the Physical Readiness Test 
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2. Training Topics 


Training and enculturation is the focus of Plebe Summer. Therefore, there are 
specific Naval Academy instructions that schedule, guide, and control the training and 
cultural indoctrination process (USNA 1996; USNA, 2001la; 2001b, USNA, 2001If; 
USNA, 2001g). There are eight focus areas of Plebe Summer training (USNA 2001a). 
Focus areas are distinguished from one another by distinct objectives. Each focus area 


has discrete lesson topics associated with its requisite objectives: 


¢ Command Religious Programs is an introduction to worship opportunities at 
the Academy and provides time for midshipmen to meet with Academy 
chaplains. Additionally, chaplains meet with midshipmen in small groups to 


discuss issues such as adjustment to USNA and interpersonal relationships. 


e Physical Education training is comprised of two parts, physical conditioning, 


testing, and training and introduction to various sports opportunities. 


e Academic focus area provides midshipmen with an introduction to their 
academic careers at the Academy. It includes a series of placement exams, 


advising sessions, and curriculum briefs. 


e Supply objectives include all equipment and gear logistics. Therefore, Supply 


is referred to as activities rather than training or instruction. 


e Administrative focus area provides time periods to fill out all required 


paperwork and general administration following induction. 


e Medical/Dental provides time for initial examinations. Additional training is 
provided in first aid, smoking cessation, drug and alcohol abuse, and personal 


hygiene. 


e Professional Development is designed to introduce midshipmen to various 
professional skills. These include sailing, damage control, career information, 
and visual signals training. Midshipmen are provided training in three levels 


of small sailing craft; knockabouts, lasers, and Navy 44’s. 
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¢ Commandant of Midshipman time is the largest and most diverse focus area. 
It serves as a “catch-all” for topics not associated with the previous seven 
focus areas. Topics covered in this focus area include Honor Training, 
Military History, Close-order drill, character development, and others. 
(USNA, 2001a, p. 4-20) 


Appendix A lists the lesson topics and activities for each focus area. Topics in each 
focus areas are presented to midshipmen in m assortment of formats also listed in 


Appendix A. The following is a description of training formats: 


e Instruction (Inst) is a period of training for which specific terminal training 
objectives exist. Upon completion of an instruction period, plebes will have 


gained a new skill or knowledge that is testable at some future time. 


e Brief (Brief) is a period of instruction for which no terminal training 
objectives exist. Although information gained will be useful at some future 
time, information is not testable. Examples are Superintendent and 


Commandant of Midshipmen briefs. 


e Administrative Event (AdmnEvnt) is a period of time required for 
administration of regimental requirements such as haircuts and equipment 


issue. No training objectives are associated with these events 


e Operational Events (OpEvnt) are required for regiment-wide operations. 
Examples are visits to the Holocaust Museum and Baltimore Orioles baseball 


game. No training objectives are associated with these events. 


e Examinations (Exam) are used to determine skill or knowledge level of 
midshipmen. These time periods are used to evaluate training effectiveness. 


(USNA, 2001a, p. 3) 


The majority of training is in the form of Squad Leader Instruction (SLD periods. 
Plebes receive twenty-two hours of SLI as outlined in Appendix B. These instructional 
periods present Plebes with information to facilitate the transition from civilian to 


military life (USNA, 2001a, p.53). 
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3. Plebe Performance Evaluation 


Plebe development is measured through written examination, verbal examination, 
inspections, physical fitness tests, and practical application. Plebes must pass written 
examinations in the subjects of nautical rules of the road, damage control, and visual 
signaling. A score of 70% is required to pass written exams. Verbal examination are 
conducted in the form of “rates”. Plebes are assigned professional knowledge that must 
be acquired according schedule. They are verbally quizzed (rated) by instructors to 
assess their level of knowledge. If plebes fail to demonstrate assigned professional 


knowledge they receive additional training and are immediately re-tested. 


Plebes are inspected often during Plebe Summer. Primarily inspections are 
conducted on uniform appearance and room cleanliness. Failure in amy inspection 


receives immediate remediation and correction. (USNA, 2001b, p. 5-31) 


Physical fitness training and testing are a large part of Plebe Summer. The 
Physical Readiness Test (PRT) is conducted three times during Plebe Summer; each 
plebe must participate in an initial, intermediate, and final PRT. The PRT consists of 
three elements: 1) number of pushups in two minutes, 2) number of sit-ups in two 
minutes, and 3) timed 1 % mile run. Plebes failing to pass the initial or intermediate 


PRT’s receive remediation throughout Plebe Summer (USNA, 2001b, p.22). 


Practical application of training received in weapons, sailing, and visual signals 
are required during Plebe Summer. Plebes receive the opportunity to qualify in small 
sailing craft and weapons (USNA, 200la, p. 32) Additionally, Plebes must qualify a 
professional watch, the Midshipman Mate of the Deck. 


Plebes receive formal evaluations from midshipman detailers twice during Plebe 
Summer. Counseling sheets are retained by squad leaders to record any item considered 
significant in the plebe’s development (USNA, 2001a, p 5-25). Upper-class midshipmen 
provide evaluation of the Plebes in the following areas: personal appearance, military 
bearing, attitude, professional knowledge, self-confidence, watch standing, military drill, 
room appearance, and conduct. Plebes receive a subjective grade of excellent, average, 


or below average in each area. Also, academic and military performance tests are 
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recorded on plebe evaluations. In areas where Plebes are evaluated as below average, 
remediation is provided (USNA, 2001a, p. 5-25). 
E. CHAPTER SUMMARY 


This chapter has reviewed current literature addressing performance measurement 
and management and its potential benefits in assisting organizations to change in the face 
of diverse cultural and technological advance. The purpose, goals and objectives, best 
practices, and terminology of performance management were reviewed. The chapter also 
reviewed the 1993 Government Performance and Results Act and how the act 


institutionalized performance management in the federal government. 


Although most literature provides positive support for performance measurement 
and management, the chapter reviewed some possible drawbacks and pitfalls to 
performance management that organizations must take into account performance 


measurement programs are developed. 


The United States Naval Academy is subject to the GPRA and has developed its 
own strategic plan. The plan was introduced in this chapter as well as its purpose, 


objectives, and goals. 


Finally, the Naval Academy’s Plebe Summer program was introduced. The 
introduction included the purpose, goals, objectives, and training curriculum of plebe 
summer as well as Plebe evaluation. The remaining chapters of this thesis will more 
deeply examine how the Naval Academy uses performance management to evaluate the 


effectiveness of the Plebe Summer program in meeting its objectives. 
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Il, PERFORMANCE MEASUREMENT MODEL 


A. INTRODUCTION 


There are numerous performance measurement models. This chapter describes 
Hatry and Kopczynski’s (1997) Guide _to Program Outcome Measurement for the 
Department of Education This model provides procedures and recommendations for 
developing an outcome measurement process for individual educational programs (Hatry 
and Kopczynski, 1997, p. 1). The focus of the model is on developing a system that 
identifies and measures outcome indicators rather than other performance measures such 
as goals, outputs, or processes: 

Without information on results, managers can only supervise “inputs” or 

monitor processes. Decisions on whether the program is actually working 


well — or what needs changing — are made in the absence of hard data on 
actual outcomes (Hatry and Kopczynski, 1997, p. 1). 


Furthermore, the philosophical premise of the model is that outcome measurement has 


four basic applications (1997, p. 2): 


e First and foremost outcome information should help program managers and 
their staff track how their programs are performing. That information, in turn, 
assists program managers and guides improvement efforts. The information 
should, for example, indicate where, when and under what conditions 


outcomes appear to be satisfactory or not satisfactory. 


e Second, outcome information can be useful for developing and justifying 
budgets and for formulating recommendations as to needed legislation and 


policy. 


e Third, outcome information is used by governing organizations in establishing 


accountability of programs for program quality and outcomes. 


e Lastly, outcome information can be used to help communicate with and 
inform customers and the public at large as to the extent to which progress is 


being made. 
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The model’s structure is simple. It consists of twelve steps divided into two 


categories: three preliminary steps and nine process/implementation steps. The intention 


of the model is to assist managers in developing high quality performance measurement 


systems or improve the ones already in place (1997, p. 2). 
B. PRELIMINARY STEPS 


Three preliminary steps should be taken prior to commencing the tasks of 


developing an outcome measurement process (Hatry and Kopczynski, 1997, p. 11-12). 


1. 


Determine the scope of the program included in the measurement process. If 
there is more than one activity associated with the program then program 
managers must determine which activities will be included in_ the 
measurement process. Moreover, many programs contain activities that 
overlap with other programs. Managers should decide, prior to developing a 
measurement process, which overlapping programs will included in their 


measurement processes 


Secure top-level support for the outcome measurement effort. Strong 
commitment is necessary from senior level managers in order to insure 
adequate resources are committed to develop, implement, and operate the 


measurement process. 


Establish a working group to oversee development of the process. The 
working group should include stakeholders from every aspect of the program. 


Along with the program manager, important stakeholders include: 
a. Staff and other employees 
b. Representatives from related programs 
c. Technical Experts or outside consultants 
d. Budget personnel 


e. Program users 


These preliminary steps ensure a strong foundation is constructed prior to building an 


outcome measurement process. Program managers should not omit these preliminary 
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steps because they provide the initial framework for the development and implementation 


processes. 


c. DEVELOPMENT AND IMPLENTATION STEPS 


The following nine steps detail the necessary actions program managers must 
undertake and then use to develop and implement a successful outcome measurement 


system (Hatry and Kopcezynski, 1997, p. 19-87). 


1. Identify program mission/objectives and customers: The first step is precisely 
identifying the program mission, objectives, and customers. It begins with 
developing a mission statement that will serve as a starting point for 
identifying the outcomes to be measured and the specific performance 
indicators required. The mission statement should describe the overarching 
results the program is designed to achieve or the purpose of the program and 


have the following characteristics: 


a. General statement of the major missions/purposes/objectives that the 


program is intended to achieve 


b. The statement should not contain numerical targets. Such targets will 


be developed later in the development process. 


c. All major objectives that the program hopes to achieve should be 
identified 
d. The statement should include words that explicitly call for minimizing 
possible negative or unintended program effects. 
Customers should be identified unless this is obvious. Questions that should be answered 
when identifying customers include: 1) who benefits from the program? and 2) who 
might be hurt by program activities? 
2. Identify outcomes that should be monitored. All relevant outcomes should be 


listed. There are numerous sources that identify important outcomes and are 


often are the same as the sources that identify program mission and objectives. 


a. Legislation 
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b. Mission statements 

c. Strategic Plans 

d. Direction from senior leadership 

e. Direction from individual customers or focus groups 
f. Individual employees or focus groups. 

g. Complaint information 

h. Other similar programs 

i. Logic models or outcome sequencing charts. 


3. Select outcome indicators: Program outcomes are not the same as outcome 
indicators. Every listed outcome should be translated into one or more 
outcome indicators that identify specifically what is to be measured. Outcome 
indicators identify specific numerical or qualitative value that indicates 
progress toward or away from achieving an outcome. Good quality indicators 


meet certain criteria: 
a. Relevance to mission and the outcome it is intended to measure. 


b. Importance of what it measures. Does it measure some important 


aspect of the outcome? 
c. The extent it is not duplicated by other indicators. 


d. The extent that program has influence or control over the values of the 
outcome. Although programs cannot achieve complete control over an 
outcome, as long as the program is expected to ultimately have some 
tangible, measurable effect on the outcome, the indicator should be 


considered for inclusion. 


e. Feasibility and cost of collecting indicator information. It should be 
noted that sometimes the most costly indicator provides the best 


quality data and should be retained. 
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f. Indicators must include a numerical statement such as “percentage”’, 


99 66 


“incidence”, “proportion”, or “rate of”. 


g. Level or ability of customers to provide relevant feedback relating to 


outcomes. 


h. Wording of indicators must be specific. For example, a program that 
wants to increase the “number of teachers that have received 
significant professional development’ will need to define specifically 
what is meant by “significant” in order to be able to measure the 


outcome indicator in a meaningful way. 


4. Identify data sources and data collection procedures: The list of indicators is 
not complete unless data collection methods are included. Four major sources 
of educational outcome data include agency records, administered tests, 
customer surveys, and trained observer ratings. These four sources each have 
strengths and weaknesses when serving as data collection methods. 


Furthermore specific requirements to ensure each source is effective are listed. 


Agency records are a good source of data due to their accessibility and low cost. 
Moreover, the procedures associated with record keeping are usually familiar to program 
personnel. However, modifications to existing record-collection processes may be 
required to obtain usable outcome data. Additionally, agency records alone seldom 
provide enough information on program quality or outcomes and, sometimes, record 
information may have to be merged with information derived from the records of other 


programs. 


Student assessment through performance testing is a valuable source of outcome 
data. They are excellent indicators of program performance and outcome data. 
Moreover, grouped test data can be categorized by various breakout characteristics thus 
making the data more exploitable. Testing data should, however, be collected in a timely 


fashion so that program outcome is best represented by the data. 


Customer surveys are an important source of data. Surveys of customers, 
systematically conducted, are a major way to obtain information on outcomes such as 
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behavior and satisfaction with program characteristics. This information often is 
unavailable from any other source. Moreover, surveys provide direct input from 
customers that add credibility to collected data. However, surveys are difficult to design 
and often expensive. Expertise is required when developing a survey instrument in order 
to ensure accuracy and relevancy of data gathered. Additionally, surveys are based on 
respondents’ perceptions and memory. ‘These weaknesses can cause data to be less 


convincing than other data source. 


Surveys designed to collect data on outcome measurement should include 
questions relating to outcomes of services, amount of service that respondents used, 
diagnostic questions (why respondents gave particular answers), demographic questions, 


and requests for improving services. 


Trained observer ratings can be used to measure outcomes observable by the 
physical senses. With this method personnel are trained to make qualitative ratings of 
program outcomes. This method can be applied in situations where the outcome can be 
measured by physical observation and can be rated on a scale that identifies variations in 
condition. These observations can have a high degree of accuracy and reliability if the 
rating system is clearly defined, adequate training of observers is conducted, and if 


procedures for periodic quality reviews are developed and implemented. 


Like the other methods of data collection, there are advantages and disadvantages 
to this method. The advantages of trained observers include reasonably accurate rating of 
conditions that are otherwise difficult to measure, attaining data that can be used to 
allocate resources, and observer reports are usually easy to present and understand. 
Disadvantages include the labor intensive method, ratings require monitoring to ensure 


observers adhere to standards, and personnel may feel uncomfortable with the method. 


Outcome indicators should not be considered final until data sources and data 
collection procedures are chosen. Data collection procedures can affect the specific 
outcome indicator used to measure a particular outcome. Program managers may 
consider using more than one procedure and thus more than one indicator to track 


important outcomes. 
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a Select outcome indicator breakouts: Program data alone does not provide 
managers with usable information; intelligent program decisions cannot be made using 
raw data. Therefore, all data collected should transformed before being used as a 


decision tool. 


The first transformational step is disaggregation of data (or selecting breakouts). 
Breakouts permit comparisons among groups within a program. They also should be 
used to distinguish important groupings that have quite different outcomes from other 
groups. Therefore, outcome analysis should track the progress being made separately for 
each group to provide more meaningful information on what is happening within the 


program. 


Programs should consider a number of categories when deciding what breakouts 
are most useful for its outcome indicators (Hatry and Kopczynski, 1997, p. 53). 
Categories of breakout information include: organizational or unit objectives, customer 
characteristics, and type of process used to deliver the service. These categories enable 
program managers to separate data in order to identify where their programs are doing 


well and where they are not. 


Separate outcome information on individual projects within larger programs can 
be quite useful. When data is collected from large programs managers of individual 
projects should have outcome information that pertains to that manager’s own area of 
responsibility. Outcome data that lumps together outcomes from more than one project is 
not likely to be useful to managers of individual projects. Thus, program objectives 
should be identified prior to data collection so that breakouts can be more easily 


identified in the later process. 


Breakouts by categories of customers can be very useful in providing information 
to program personnel about the extent to which particular categories of customer services 
are achieving the desired outcomes and for which categories desired outcomes are not 
being achieved. Categories in this breakout include: age, gender, race, household 
composition, etc. Using this information, breakout characteristics can be 


comprehensively tailored. 
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Relating outcomes to the type and magnitude of activities being supported by the 
program is likely to be of major interest to program managers. Thus, a program should 
consider breaking out outcome data by key characteristics of the projects being supported 
by the program. Such information can be very useful in distinguishing the more 


successful from less successful approaches. 


Many programs have special characteristics of relevance to them that do mt fall 
into one of the previous categories. Therefore, they should examine each of its outcome 
indicators to determine what breakouts would likely help them identify where the desired 


outcomes are successful and where they are not. 


Breakout selection should start with a review of the categories listed and a review 
of outcome indicators to determine which apply to the program. Sub categories should 
be identified within each major category. For example, by what age level groupings 
should be measured? Under 20, over 25, between 17 and 22? Or should data be sought 
by individual ages? Which subgroups and how many will be sought will be determined 


in part by: data collection procedures, available resources, and required accuracy. 


6. Compare findings to benchmarks: Benchmarks provide a means to 
determine if program outcomes are good or bad. This step identifies the types of 
comparisons (benchmarks) useful for programs. This step is a major one in a program’s 
analysis of the findings from its outcome measurement data collection. Major types of 
comparisons include: Previous performance that may indicate improvement over time, 
performance of similar units that may provide benchmarking against the best, pre- 


selected targets, and outcomes among different client groups. 


Comparisons against previous performance are the most common type of 
benchmark. Current performance should always be compared to that of previous 


reporting periods, regardless of the data collection cycle. 


Programs that provide essentially the same service are likely to provide useful 
benchmarks to managers. Reporting such comparisons can also have motivational value 


to program personnel. 
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Breakouts by various customer demographics provide ideal benchmarking data. 
Comparisons should be made to indicate whether the program appears to be more or less 
successful with certain categories of customer/workload than with others such as males 


compared to females (Hatry and Kopczynski, 1997, p. 62). 


Pre-selected targets (goals) are extremely useful benchmarks. They are 
cornerstones of the GPRA. Programs set targets and report achievement or non 
achievement. Programs may want to use range targets rather than single values. For 
example if a target is expressed as a percentage, the target might be the range 


encompassing the target plus or minus five percentage points. 


When an alteration of program delivery occurs outcome measurement can 
indicate the success of the alteration. This benchmark uses previous performance as a 
standard to measure new procedures. Outcome data can be used to track changes in 


outcomes from before the change to the outcomes after introduction of new practices. 


yi Pilot test the procedures: New or substantially altered data collection 
procedures should be pilot tested prior to full implementation. This will identify and 


eliminate any problems with data collection procedures. 


Testing should look for problems such as: unclear definitions, missing data, and 
confidentiality problems. Resolution of these problems can include procedure alteration, 


deletion of outcome indicator, or acceptance of diminished accuracy. 


8. Analyze and report outcome information’ Steps 5 and 6 discussed 
breakouts and comparisons for each outcome indicator. These choices of breakouts and 
comparisons will likely be the major focus of a program’s analysis effort after it receives 


outcome information. 


In each comparison developed in step 6, program staff should identify outcomes 
that indicate where the program has done considerably better or worse than anticipated. 


Furthermore, staff should attempt to identify why this occurred. 


How programs report can be as important as what they report. There are 


numerous methods to report outcomes: 
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a. Present actual outcomes to targets for both the last and current 


reporting periods. 

b. Present cumulative values for a year rather than previous reporting 
periods. 

c. Present responses from customer surveys with responses broken out by 
respondent characteristics. 


d. Other methods such as graphs, bar charts, timelines, and maps. 


When outcomes are reported explanatory information should be included. Such 
information gives the program the opportunity to explain significant outcomes that were 


better or worse than expected. 


9. Use outcome information: It is important to remind outcome information 
users that outcome data don not tell how much of the observed outcomes was caused by 
the program hand how much by other factors. Outcome information only conveys 
whether programs are meeting outcome objectives or not. The primary use of outcome 
information is to provide regular feedback to program personnel. There are various 
applications to the feedback of outcome information: 

a. Motivate personnel to improve. 

b. Track whether actions have led to improvements 

c. Identify where problems exist 

d. Communicate with and inform customers and the public about the 
extent to which progress is being made. 

e. Support long-range planning. 


f. Serve as baseline for targets and goals. 


g. Identify need for program personnel training. 
D. CHAPTER SUMMARY 


This chapter reviewed Hatry and Kopczynski’s (1997) Guide to Program 


Outcome Measurement for the U.S. Department of Education The guide provides 
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recommendations for development of an outcome measurement process for individual 


education related programs. 


The first three steps are preliminary to process development. These preliminary 
steps are intended to gather high-level program support, determine the scope of the 
program to be measured, and establish a working group to oversee the development of 


the process. The guide then recommends nine process and implementation steps: 
e Mission, objective, and customer identification. 
e Identify outcomes to be monitored 
e Select outcome indicators 
e Identify data sources and data collection procedures 
e Select outcome indicator breakouts 
e Set benchmarks and compare findings to benchmarks. 
e Pilot test data collection procedures 
e Analyze and report outcome information 
e Use outcome information 


Outcome measurement processes developed using this guide provide program 
managers with information that will assist them in tracking the performance of their 
program and identify where improvements are required. Moreover, outcome information 
will assist managers in communicating with and informing customers and the public as to 


the success of education related programs. 


The remaining chapters of this thesis will describe the research methodology, 
analysis, conclusions, and recommendations for the Naval Academy’s Plebe Summer 
program performance measurement process using Hatry and Kopczynski’s guide as a 


basis of analysis. 
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IV. METHODOLOGY 


A. INTRODUCTION 


The purpose of this research is to analyze the performance measurement and 
outcome management procedures applied to the Plebe Summer program at the United 
States Naval Academy. The research methodology is divided into four areas. The first 
area consists of a thorough examination of current performance measurement and 
outcome management practices. The Government Performance and Results Act of 1993 
forced United States governmental agencies to apply performance measurement practices 
in planning and managing their programs. The Naval Academy is part of the Department 
of Defense; therefore, it is subject to the GPRA. Due to the requirements of the GPRA 
investigation of current performance measurement and performance management is the 


logical first step. 


The second area of research methodology consists of a thorough examination of 
the United States Naval Academy directives and military instructions governing the 
management of Plebe Summer. This area provides baseline information concerning the 
performance measurement mechanisms currently used by the Naval Academy. 
Moreover, this area also examines how midshipman performance is measured and 


recorded during and after Plebe Summer. 


Gathering feedback from expert Naval Academy personnel is the third area of 
research methodology. In order to effectively identify the precise methods of Naval 
Academy performance measurement applied to Plebe Summer. Experts selected for this 
research included only personnel in key management positions accountable for 


determining the success of Plebe Summer. 


The final research area uses Hatry and Kopczynski’s (1997) Guide to Program 
Outcome Measurement to evaluate the performance measurement and outcome 
management practices applied to the Plebe Summer program by the Naval Academy. 
This guide proposes nine development and implementation steps in creating an effective 


performance measurement system for educational programs. 
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B. RESEARCHING GENERAL PERFORMANCE MEASUREMENT 


Researching Plebe Summer performance measurement begins with an in-depth 
review of current performance measurement literature. This management tool has 
applications in both private and public sectors. Therefore, not all theories apply to the 
Academy’s Plebe Summer program. An extensive literature review concentrates on 
current theories and best practices applicable to public sector programs. This review 
examined books, journal articles, internet articles, encyclopedias, and governmental 
agency reports. The following subjects concerning performance measurement are 
researched in order to identify how best to apply performance measurement to the Plebe 


Summer program: 


e Goals and objectives of performance measurement are studied in order to 
confirm the Plebe Summer program is indeed an activity with which 


performance management is applicable. 


e Performance Terminology is researched to provide a consistent vocabulary 


throughout the thesis. 


e Best practices of performance measurement are researched to provide positive 
examples of public sector employment of performance management. This 
area identifies how public sector programs work to meet the demands of the 


Government Performance and Results act of 1993. 


e Misuses of performance measurement are researched to identify areas where 
performance management is misapplied and could possibly lead to poor 
management decisions 


Cc. RESEARCHING ACADEMY PERFORMANCE MEASURMENT 


Plebe Summer performance measurement procedures are derived directly from 
Naval Academy instructions governing Plebe Summer. Therefore, a detailed review of 
all instructions governing Plebe Summer is conducted. Moreover, because Plebe 
Summer instructions should support larger institutional goals a review of the Naval 


Academy strategic plan is conducted. Additionally, this strategic plan is reviewed to 
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provide information on the Academy leadership’s attitude and goals concerning the full 


development of midshipmen. 


Specific governing instructions for the Plebe Summer program are researched to 
identify the exact goals, objectives, procedures, and performance measurement of Plebe 
Summer. These instructions provide information concerning how Plebe Summer 
performance is recorded and examined in determining the level of success or failure of 


the program in meeting program objectives. 


D. EXPERT INTERVIEWS 


Expert interviews are conducted to identify the performance measurement 
procedures used by Naval Academy personnel responsible for planning, executing, and 
evaluating Plebe Summer. Furthermore, expert interviews are used to further elaborate 
the intentions of Naval Academy instructions and as well as identify the depth to which 


the instructions guide the performance measures of Plebe Summer. 


Interviews are conducted with six experts that cover three levels of Plebe Summer 
program management. Three levels of management were selected to gain a thorough 
cross-sectional view of performance measurement practices. The three levels and related 


interviews are as follows: 
e Commandant of Midshipman 
e Officer In Charge of Plebe Summer Detail 
e Assistant Officer In Charge of Summer Detail 
e Company Officers (3) 


Prior to conducting interviews, interviewees were provided a Pre- interview 
package. The package included a coversheet that introduced the purpose and intentions 
of the interview. Moreover, it provided the interviewees definitions of the performance 
measurement terms associated with this research. The purpose of providing this package 
to interviewees was to insure similar knowledge and understanding of the esearch 


model. The pre- interview package is shown in Appendix C. 
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A Plebe Summer Outcome Measurement Matrix was also included in the pre- 
interview package. This matrix provided interviewees a summary of the performance 
measurement practices defined in the Naval Academy’s instructions governing the Plebe 
Summer program. The matrix is designed using the steps specified in Hatry and 
Kopcezynski’s (1997) Guide to Program Outcome Measurement. Interviewees are asked 


to review the matrix while considering the following questions: 


e In your opinion, does this list of objectives cover all the objectives of Plebe 


Summer? If not, please list additional objectives. 
e In your opinion, what are the outcomes these objectives intended to achieve? 
e What are the indicators of each outcome? 
e How is data on each indicator collected? 


e What is the benchmark for each indicator? What are the minimum acceptable 


levels of performance associated with each indicator? 


The interviews were semi-structured and used open ended questions to give 
interviewees latitude in their answers. Questions were general and covered the following 


topics and theme: 
e What, in your opinion, is the primary purpose of the Plebe Summer Program? 
e What, in your opinion, are the goals and objectives of Plebe Summer? 
e What outcomes are associated with each objective? 


e How do you determine the success or failure of Plebe Summer in meeting its 


goals and objectives? 


e What information do you use in determining the success of failure of Plebe 


Summer? 


e Using the provided matrix please identify what, in your opinion, are the 
indicators of each outcome you believe are used to measure the success or 


failure of each goal or objective? 
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e What data do you collect and use for each outcome indicator? 
e How is outcome data recorded and reported? 


e What benchmarks do you use in order to evaluate the data collected? 


E. OUTCOME MEASUREMENT EVALUATION 


The final research area uses Hatry and Kopczynski’s (1997) Guide to Program 
Outcome Measurement to evaluate performance measurement and management applied 
to Plebe Summer. The guide provides a mechanism to assess the effectiveness of the 


current outcome measurement process applied to the Plebe Summer program. 


Information gathered from literature reviews and expert interviews are fused with 
the guide to provide a foundation for assessment. Specific Plebe Summer objectives, 
outcomes, outcome indicators, and breakouts listed in governing instructions and expert 
interviews are assessed using the guide. The purpose of the qualitative assessment is to 
determine the effectiveness of the Plebe Summer performance measurement and outcome 
management procedures. 


F. CHAPTER SUMMARY 


This chapter described the methodology for this research. The research 
methodology consists of literature reviews, expert interviews, and analysis. First, a 
review of the current performance measurement and outcome management literature is 
conducted. Next a thorough review of Naval Academy instructions governing Plebe 
Summer follows. Interviews of six Plebe Summer experts are conducted to identify the 
performance measurement procedures used by personnel responsible for the planning and 
conduct of Plebe Summer. The following chapters will analyze the Plebe Summer 
performance measurement and outcome management procedures then provide 
conclusions, recommendations to the Naval Academy, and recommendations for further 


research. 
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Vv. DATA ANALYSIS 


A. INTRODUCTION 


This chapter analyzes the data collected from the review of Naval Academy 
instructions and interviews of Plebe Summer experts. Each Plebe Summer objective is 
examined to evaluate the performance measurement and outcome management practices 
applied to Plebe Summer by the Naval Academy. 

Developing program outcome information is a critical step in developing a 

high quality performance measurement system for programs and projects 

in the federal government. Good program management requires collection 

and use of outcome data to provide guidance for improvement. Without 

information on results, managers can only supervise “inputs” or monitor 

processes. Decisions on whether the program is actually working well — 


or what need changing — are made in the absence of hard data on actual 
outcomes (Hatry and Kopczynski, 1997, p.1). 


This analysis determines whether Naval Academy employs performance measurement 


and outcome management practices effectively to achieve Plebe Summer objectives. 


The Guide to Program Outcome Measurement consists of nine process and 
implementation steps and is intended to be a tool used by managers and their staffs in 


building reasonably valid and reliable outcome monitoring procedures. Evaluating the 
outcome management procedures applied to Plebe Summer divides these nine steps into 
two areas. First, this evaluation begins by identifying the objectives of Plebe Summer. 
This establishes a foundation to answer the question, “What is successful performance of 


Plebe Summer?” (Hatry and Kopczynski, 1997, p.1) 


The second step is to evaluate the current Naval Academy performance 
measurement and outcome management procedures for each identified objective. Plebe 
Summer objectives are subsequently evaluated independently using the following 


criteria: 


e Outcomes (O). Are specific outcomes for each objective identified? Do 


outcomes cover each element of the objective statement? 
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e¢ Outcome Indicators (OI). Are outcome indicators that specifically identify 
what is to be measured identified? Do indicators 1) measure an important 
aspect of the outcome, 2) overlap with other indicators, 3) cover all outcomes, 
and 4) provide relevant feedback? Are indicators controllable by the 


program? 


e Data collection (DC). Are reasonable data sources available? Are data 
collection procedures for each indicator identified? Does data collected allow 


for comparison? 


e Benchmarks (BM). Are benchmarks that provide the ability to assess the 


successful accomplishment of each objective identified? 


To be characterized as effective the performance measurement procedures and outcome 
management procedures of each objective must meet these criteria. 


B. PLEBE SUMMER OBJECTIVES 


The first step in developing an outcome monitoring process is to prepare a 
mission/objectives statement for the program. This identifies what overarching results 
are hoped to be achieved or the purpose toward which the program is directed (Hatry and 
Kopcezynski, 1997, p.17). Identifying program objectives is the starting point for 


identifying outcomes and performance to be measured later. 


The basic form an objective statement should include both a “To” statement and a 
“By” statement. The “To” statement identifies the basic objectives the program seeks. 
The “By” statement identifies the method by which the objectives are accomplished 


(Hatry and Kopczynski, 1997, p.18). 


Plebe Summer is managed using the following military instruction: Fourth Class 
Indoctrination (Plebe Summer) COMDTMIDINST 1510.2B (USNA, 2001b). This 


instruction lists three Plebe Summer objectives: 


e Effect the transition from civilian or enlisted to officer candidate status. 


52 


e Indoctrinate and train midshipmen in proper military discipline, courtesy, 
requirements and traits of special trust and confidence, teamwork, and 


personal accountability. 


e Provide basic training in seamanship, marksmanship, military drill, watch 
standing, and customs and traditions of the Naval service (USNA, 2001b, p. 
6). 


Expert interviews identified one additional Plebe Summer objective not identified 


in Naval Academy instructions: 


e Prepare the Plebe class for integration/incorporation into the Brigade of 


Midshipmen and the challenge of Plebe academic year. 


Although this last objective is not addressed by Naval Academy instruction it was 
a universal theme among interviewees. Commander Patricia Cole, USN expressed this 


objective saying: 


To prepare the incoming class with the rudimentary knowledge and skills 
so that they are equipped and prepared to be incorporated into the Brigade 
of Midshipmen. They are in an indoctrination process that goes 
throughout the entire year and this is just a part of it. Plebe summer is to 
get them into physical conditioning and teach them the basic military 
skills and knowledge about the Naval Academy and their responsibilities 
and expectations. (Personal Communication, April 2002) 


The Commandant of Midshipmen, Colonel John Allen, USMC amplified this idea 
saying: 
In a very real sense the focused purpose of Plebe Summer is, through an 
environment of imposed discipline, to transition them from individual 
priorities to collective and small unit priorities associated with institutional 
excellence as a member of the fourth-class regiment. And, then have them 


transition out of the fourthclass regiment and into the Brigade of 
Midshipmen. (Personal Communication, April 2002) 


These four objectives provide adequate “To” statements. However, the Naval 
Academy instructions governing Plebe Summer do not contain a “By” statement. The 


instruction that governs Plebe Summer provides limited clarity on how objectives are to 
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be met when it defines the scope of Plebe Summer. This definition provides nebulous 
direction to the methods employed by Plebe Summer: 
Plebe training encompasses progressive, demanding training in those 
subjects required to produce midshipmen who function effectively in the 
brigade, are trained in drill, are physically fit, and practice those personal 


and professional traits which distinguish them as midshipmen at the 
United States Naval Academy (2001b, p. 6). 


This scope declaration is vague and incomplete. It does not define the terms, 
“progressive” and “demanding” or define what “personal traits” distinguish plebes as 
midshipmen. The “By” statement should lead management personnel to exactly what 
should be measured by the performance measurement system. However, in the case of 
Plebe Summer, no clear guidance is provided. The absence of a clear and direct 
statement associated with each objective leads to misinterpretation of the objectives of 
Plebe Summer. This misinterpretation was expressed by Naval Academy experts when 


they discussed the objectives of Plebe Summer: 


e There are all kinds of factors that I’m not sure we, as an institution, 


have properly or accurately articulated (Colonel John Allen, 
USMC, Personal Communication, April 2002) 


e There is a sense that plebe summer without a high attrition rate 
wasn’t really a Plebe Summer. There is the sense that Plebe 
Summer is a stand-alone event, an experience. None of that is 
right. There will be some attrition during Plebe Summer, but the 
intent of Plebe Summer is not attrition. (Colonel Allen, Personal 
Communication, April 2002) 


e During Plebe Summer, I think there is a misconception by some of 
the detailers that they are preparing the plebes for battle, preparing 
them for war, or trying to train them to be these people who finish 
plebe summer and get immediately flown over to combat 
terrorism. Which isn’t the case. (Captain Tyrel Moxey, USMC, 
Personal Communication, April 2002) 


A complete mission/objective statement is important in developing a valuable 
performance measurement and outcome management system. It is the starting point for 


identifying the outcomes to be measured and the specific performance indicators that are 
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needed (Hatry and Kopcezynski, 1997, p. 17). The absence of a “By” statements for these 


four objectives forces Academy personnel to develop ad hoc performance indicators. 


Notwithstanding the lack of a “By” statement these four objectives provide a 
foundation for analyzing Plebe Summer performance measurement and outcome 
management procedures. Each address one aspect of the purpose toward which Plebe 
Summer is directed. Although there is no “By” statement associated with the objectives, 


the objectives do expose the ends toward which Plebe Summer is designed to achieve. 
c. OUTCOME ANALYSIS 


The remainder of this chapter evaluates the performance measurement and 
outcome management procedures for each of the above four objectives. The following 
criteria are used in assessing the Plebe Summer performance measurement procedures 
identified during the literature review and expert interviews: outcome identification, 
outcome indicators, data collection procedures, and benchmarks. The outcome and 
performance measurement procedures should assist Plebe Summer leaders in focusing on 


results, not just work activities. 


There is a hierarchical pattern to the four listed objectives. The first objective is 
focused on the cultural indoctrination of plebes. The second objective is focused on 
furnishing plebes with the tools to be successful within the Academy’s unique culture. 
The third objective is intended to provide plebes with necessary skills. Finally, the last 
objective is intended to insure plebes are prepared apply these tools and skills as they 
enter the Brigade of Midshipmen. Thus, the hierarchy of objectives applied to Plebe 
Summer can be listed as culture, tools, skills, and application. 


1. Civilian To Enlisted Transition 


The first objective evaluated is the transition from civilian and enlisted to officer 
candidate status. This objective is focused on immersing plebes in the Academy’s culture 
and inculcating plebes with the cultural norms. Table 1, the Civilian to Officer Candidate 
Transition worksheet displays the results of the analysis. Along the top criteria categories 
are listed. Then, each row defines an outcome, lists outcome indicators, states whether 


the criteria are met, or indicates whether the measurement procedures for that outcome 
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indicator are effective. First the outcomes are evaluated; then each criterion is evaluated 
and explained in detail. 


a. Outcome Identification 


This objective is very weakly defined. Although the objective seems clear 
and concise at first glance, it contains no definition of “officer candidate status”. Naval 
Academy experts agree that, to them, this objective is linked to a cultural indoctrination 
process that occurs during Plebe Summer. Captain Paul Funk, USMC, described the 
focus of this cultural indoctrination process: 

To instill in this incoming plebe, this civilian, a heightened sense of and 
awareness to duty. It’s hard to quantify. To install a shared set of values 
among the plebes that are embraced by the brigade. Duty, honor, courage, 
commitment, ship-shipmate-self. To guide that transition and it’s changes. 


Then you see the plebes actually embrace them and understand them. 
(Personal communication, April 2002) 


Lieutenant Colonel Joseph Dauplaise, USMC, Plebe Summer Officer-in-Charge, supports 
this statement: 

We are taking a raw recruit right out of high school that has an academic 

and physical background and no known character flaws and bring them 

into this culture. We now have to transition them. We have to break them 

into a routine where you get up early and stay up late. So, we are breaking 


the civilian out of them and putting the military spin on them. (Personal 
communication, April 2002) 


There is no Naval Academy instruction that defines a specific outcome to 
be measured for this objective. However, academy experts reported that an outcome (O) 
associated with this objective as: increased military performance during Plebe Summer. 
This outcome includes professional military, physical, and academic aspects of plebe 


performance. 


This outcome successfully encompasses the entire cultural basis of 
midshipmen by focusing on all three aspects: professional military, physical, and 
academic performance. Midshipmen are college students pursuing an academic degree 
while simultaneously living within a military culture that includes values, dress code, 
military courtesy, vocabulary, and personal discipline. The combination of these aspects 
creates conditions somewhat uncommon among other educational institutions and is 


56 


important when addressing the cultural indoctrination of plebes. In the absence of 
direction provided by governing instruction, “increased military performance during 


Plebe Summer” is a suitable outcome for analyzing this objective. 


Table 1. Civilian to Officer Candidate Transition Worksheet 


Outcome Data Collection Benchmark Effective/ 
Proc dures Non Effective 


1. Increased Military 1. Military Performance N (Individual only) 
Performance during ee 
Pishe Summmce 2. N (Individual only) 
3. PRT Scores 3. N (Individual only) 
4. Time MGT Skill ‘ 4. None 
5 


2. Rate Knowledge 











5. Conduct/Honor . None 





b. Outcome Indicators 


Specific outcome indicators (Ols) for this objective that support 
performance measurement of Plebe Summer are not listed by Naval Academy 


instructions. However, Naval Academy experts list Ols they apply to this outcome: 


e Percent receiving passing military fitness report grades during Plebe 


Summer 


e Percentage passing Physical Readiness Test (PRT) during Plebe 


Summer 
e Percentage passing rate knowledge quizzes during Plebe Summer 


e Incidence of demonstrated poor time management skills during Plebe 


Summer 
e Incidence of conduct or honor infractions during Plebe Summer 
e Rate of attrition during Plebe Summer 


Each of these indicators measures important aspects of the outcome. 
Fitness report grades provide an accurate description of midshipmen performance 
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(USNA, 2001b, p. 5-25). Rate knowledge quizzes are designed to measure plebe 
understanding of “their individual responsibility to further his/her training and 
professional development and to facilitate the smooth and efficient operation of the 
organization itself’ (USNA, 2001b, p. 2-3). Moreover, fitrep grades directly represent 
the level to which individual plebes perform professionally. PRT scores are important 
due to the physical requirements placed on midshipmen. All midshipmen are subjected 
to a demanding schedule that requires excellent time management skills. Midshipmen 
are expected to act and conduct themselves as public figures. Therefore, incidence of 
honor and conduct offenses are used to indicate the level to which midshipmen meet this 
requirement. Finally, the attrition rate is keenly scrutinized because it indicates the level 


of plebe success in making the transition into the Academy’s culture. 


These Ols overlap in some instances. Due to the demanding schedules to 
which midshipmen are subjected rating and professional grades can be, in some 
instances, due to poor time management skills. Attrition can be affected by conduct or 


honor offenses. 


Each indicator provides relevant feedback to Plebes and Naval Academy 
leaders on the success of meeting this objective. More importantly, most of these 
indicators are very easy to understand and apply. Problems developing time management 
skills are more difficult to identify due to many ways poor time management manifests 
itself. However, when time management is identified as a source of difficulty for the 
plebe, the resultant feedback is extremely relevant and useful. Attrition rates 
significantly higher than average indicates, to Academy leadership, that the plebe class is 


having difficulty adjusting. 


Programs should have some control over outcome indicators for the 
indicators to be valid. However, this criterion should not be over-valued. As long as the 
program is expected to ultimately have some tangible measurable effect on the outcome, 
it should be included (Hatry and Kopczynski, 1997, p.31). Plebe Summer is successful in 
providing some measure of control over each indicator. Plebe Summer experts agreed 
that successful introduction to academy culture results in higher performance grades both 


academic and military. Also, PRT grades rise due to the demanding physical 
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requirements of the Plebe Summer curriculum. Additionally, Plebe Summer experts 
maintain that time management, honor, and conduct problems decrease as plebes accept 
the tenets of personal discipline that are alo an integral part of the Plebe Summer 
curriculum (Capt Tyrel Moxey USMC, Personal Communication, April 2002; Colonel 
John Allen, USMC, Personal Communication, April 2002; Lieutenant Joseph Deauplais, 
USMC, Personal Communication, April 2002). 

c. Data Collection Methods 


Plebe Summer instructions do not stipulate specific data collection 
procedures in support of performance measurement. However, data is collected for each 
OI with the exception of time management skills. Again, time management skills are 
difficult to quantify and measure due to the many ways poor time management manifests 


itself. 


Fitrep grades are assigned by the squad leaders twice during the summer. 
They are then approved by the Company Officer (USNA, 2001b, p. 5-25). However, no 
comprehensive procedures are defined that would allow class-wide collection of this data. 
PRT scores, conduct/honor issues, and attrition rates are reported and recorded into the 
Midshipman Information Data System. This database is easily accessible to every 
academy leader. Moreover, the information is consistently recorded every academic 
semester. Therefore, there is no missing data when evaluating the entire class or 


individuals within the class. 


Plebe Summer grades, including fitrep information, inspection results and 
rate knowledge are established and recorded by midshipmen squad leaders. Squad 
leaders record both positive and negative aspects of performance in their Squad Leader’s 
Notebooks using the Plebe Evaluation Record sheet. The purpose of this record sheet is 
to record any item considered significant in the plebe’s development. Items recorded 
include personal appearance, military bearing, attitude, professional knowledge, self- 
confidence, physical performance, watch standing, drill performance, uniform and room 
appearance. Criteria are graded by the squad leader as below average, average, or above 


average. The importance of squad leader data collection was articulated by Captain 
Moxey, USMC: 
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We made sure that the squad leaders were writing everything down. Each 
squad leader had a binder so there was a binder for all the plebes and the 
record of counseling were put in respective plebe’s binder. Whatever it 
may be, good, bad, or indifferent counseling. The counseling that needed 
to be done on a periodic basis was put in there. On the spot counseling 
that was both positive and negative was put in there. Anything. We 
highly encouraged them to put it down. Because we continually impress 
upon them, “if you don’t write it down it didn’t happen.” You can say, 
“This person continually can’t do whatever, but if you don’t write it down 
nobody knows it and it’s gone.” 

So, we collected that information so that when we went through and 
basically had to say, “these plebes get A’s and these plebes get B’s, and so 
on” we had information to go on. Now, was it hard numbers? Did we 
have a point scale? No. It was this person has three or four great 
counseling on demeanor and discipline and punctuality. And this one here 
has been constantly corrected for dirty room, messed up uniform, there’s 
lots of documentation here, so who gets the A and whatever. It becomes 
pretty obvious (Personal Communication, April 2002). 


The information recorded by squad leaders is not shared class-wide. 
Therefore, there is no method to assemble information concerning entire class 
performance into a single database that affords comparison class-wide. Moreover, 
comparison against previous performance or pre-determined targets is impossible. 


d. Benchmarks 


No minimum standards or targets are assigned on a program-wide basis. 
Fitrep grades have minimum passing standards on an individual basis based on an A thru 
F (4.0) scale with minimum “C” grade for passing. They indicate individual performance 


and comparison against previous classes or predetermined targets is impossible. 


PRT grades are also based on an A thru F (4.0) scale with minimum “C” 
grade for passing. PRT grades are established using a 300-point system with minimum 
performance levels defined for each of the three physical tests (pushups, sit-ups, and 1.5 
mile run) each worth 100 points. These grades are easily summarized and can be used to 
compare with previous plebe classes, pre-selected targets, or different program practices 
(such as differing physical development techniques). However, no minimum class-wide 


benchmarks are set down for comparison. 
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Attrition rates also have no minimum or maximum targets. However, 
occurrence rates can easily be summarized and compared with previous plebe classes. 
The same applies for the incidence of conduct and honor offenses. Although no 
minimum standard is set, the data collected can also be compared to previous classes. All 
conduct and honor offenses are adjudicated and remediated within a short period after 
infraction on a case-by-case basis according to military guidelines. If an explanation or 
training deficiency is identified as a contributing factor to the offense, remediation is 
arranged to assist midshipman development. There are no benchmark criteria for 
measuring incidence of poor time management skills. 


e. Conclusion 


The performance measurement and outcome management procedures 
associated with the civilian or enlisted to officer candidate transition are not effective. 
There are no specific performance measurement procedures defined through Plebe 
Summer instruction. However, academy experts detailed an outcome as well as a number 
of outcome indicators and data collection procedures they use in ad hoc fashion to 


measure the success of this outcome. 


The absence of a definition of “officer candidate status” limits Academy 
personnel’s ability to accurately measure the success of this objective. Until definitions 
are identified and specific procedures are established academy personnel will continue to 


use ad hoc measures of performance. 


Although individual performance benchmarks are identified for each 
outcome indicator there are no minimum benchmarks to compare the class-wide success. 
Therefore, Plebe Summer leadership cannot accurately assess the level of success the 
program has in effecting the civilian to officer candidate transition. 


2. Indoctrinate and Train Midshipmen 


The second objective is intended to provide plebes with a number of tools they 
will employ as midshipmen within the academy’s culture. The objective states five areas 
that midshipmen receive indoctrination and training: 1) military discipline, 2) military 


courtesy, 3) requirements and traits of special trust and confidence, 4) teamwork, and 5) 
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personal accountability. This objective builds upon the first in that it is intended to 
provide plebes with the tools to effectively survive in the Academy’s military culture. 
Table 2, Plebe Indoctrination and Training Worksheet displays the results of analysis. 


a. Outcome Identification 


There is no specific outcome established for this objective. Moreover, the 
objective is poorly worded and ambiguous. The terms included in this objective 
(discipline, courtesy, team work, special trust and confidence) are not defined. This 
forces Plebe Summer experts to develop ad hoc definitions and performance metrics in 
order to measure the success of this objective. The Plebe Summer experts identify that 
the desired outcome of Plebe Summer associated with this objective is: plebes learn and 


demonstrate military professionalism and personal discipline. 


Table 2. Plebe Indoctrination and Training Worksheet 


Outcome Data Collection Benchmark Effective/ 
Procedures Non Effective 


1. Learn and 1. Conduct/Honor 1. N Undividual only) 
Demonstrate Military 
Professionalism and 

Personal Discipline 3. Inspection Results , 3. N (Individual only) 


2. Personal Observations : 2. N (Individual only) 





4. Rate Quiz results ; 4. None 





This outcome focuses exclusively on professional military aspects of plebe 
development as discussed by Commander Patricia Cole, USN: 

What I would look for is, as far as the training, do they seem to be 

handling their responsibilities well? Do they know where they are 

supposed to be? Do they know how to correctly wear a uniform? Do they 

know how to address people who are senior to them? Do they know how 


to use the chain-of-command? Basic military things. (Personal 
communication, April 2002) 


Although no specific outcome is listed Plebe Summer instruction this objective will be 


analyzed using the above outcome obtained from Academy experts. 
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b. Outcome Indicators 


Plebe Summer experts listed various indicators associated with this 


outcome. Included in the list are: 
e Low number of conduct or honor infractions 
e Incidence of outstanding scores on uniform and room inspections 
e Incidence of professionalism observed by senior personnel 
e Percentage passing rate knowledge quizzes during Plebe Summer 


These indicators focus specifically on military performance and the demonstration of 
professional military ideals. Although Plebe Summer instructions do not specify these 


indicators as metrics for this objective, they will be used for this analysis. 


Each OI measures an important aspect of the outcome. Conduct and 
honor offenses indicate understanding and acceptance of the military culture to which 
plebes are being introduced. Moreover, low incidence levels of conduct and honor 
infractions indicate high levels of personal discipline among the plebes. Room and 
uniform inspection results are indicators of plebe attention to detail and motivation 
towards maintaining high personal standards. Demonstrated professionalism such as 
proper courtesy toward senior personnel indicates comprehension of the professional 
training plebes receive as well as personal discipline. Finally, high performance on rate 
quizzes indicates plebe understanding of their personal responsibility to  self- 


improvement and professional education. 


There is overlap present among these OIs. Conduct/honor offenses are 
incorporated into military performance grades. Therefore, these indicators are duplicated 
when measuring the outcome of this objective. Also, senior personnel such as company 
officers and senior enlisted advisors have input to military performance grades. In this 


situation, their personal reflections are recorded twice. 


Each indicator listed provides relevant feedback. Furthermore, they are 
easy to understand. High rates of conduct and honor offenses quickly indicate problems 


with plebe professionalism and a high incident rate among the entire class could indicate 
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poor training of expectations. Inspection results, rate quiz results, and personal 
observations also provide the same quick indication of plebe professionalism and 


motivation. 


Plebe Summer training provides plebes with training and tools necessary 
to conduct themselves professionally within the academy culture. Although Plebe 
Summer training cannot control conduct or honor offenses it does define for plebes 
acceptable behavior and addresses the conduct expectations of each midshipmen. 
Therefore, Plebe Summer does maintain some control over these outcome indicators. 


Cc. Data Collection Methods 


Plebe military performance is observed and recorded by their squad leader 
in the same fashion describe in the first objective analysis. Squad leaders record both 
positive and negative aspects of performance using the Plebe Evaluation Record sheet. 
Conduct offenses are also recoded (USNA, 2001b, p. 71). These records are used as a 
tool when squad leaders provide performance counseling to plebes. However, the records 


of individual plebe performance are not collected for use outside the company. 


Inspection grades and rate quiz performance are also determined and 
recorded by squad leaders. Plebe performance on inspections and rate knowledge quizzes 
is recorded by squad leaders using the Plebe Evaluation Record sheet. These reports are 
not consolidated class-wide using a centralized database. Therefore, these data are not 
easily summarized and used by senior personnel. Moreover, individual and company 


average inspection performance is not recorded or tracked outside the company level. 


Incidence of professionalism observed by senior personnel are not 
recorded or tracked by anyone other than that person making the observation. No central 


data collection or recording is done with this information. 


No data are collected on “teamwork”. Moreover, Plebe Summer experts 
do not indicate understanding on how teamwork could be measured: 
Training “teamwork”? I don’t know, is there a metric for that? Is there a 
written test? There wasn’t for my company and I don’t know if you could 


create one (Captain Tyrel Moxey, USMC, Personal Communication, April 
2002) 
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An easy performance tool is knowledge and certain other standards like 
PRT scores. The harder to quantify ones are performance related such as 
“teamwork”. Or, a person’s bearing, how well they hold themselves. 
How well they react to detailers. Heck, it’s even hard to quantify if 
somebody understands teamwork or not. It’s asmall thing, but how do 
you look at someone and give them a grade on teamwork? You can’t. 
(Captain Paul Funk, USMC, Personal Communication, April 2002) 


Although teamwork can be demonstrated it is more difficult to quantify and compare 
among a group. 


d. Benchmarks 


Benchmarks that allow data comparisons are not defined. Plebe summer 
leadership use improvised methods to determine success in meeting this objective. 
Plebes are ranked using the squad leader evaluation records; however, this ranking is kept 


in-company, and no comparison among other companies is conducted. 


Fitrep grades, rate knowledge quiz performance, and inspection results 
have no minimum allowable standard. Rather, problems are identified and plebes receive 
immediate feedback on how to correct deficiencies. A further problem with inspections 
is that inspection criteria are not standardized resulting in no two plebes receiving the 
same level of scrutiny. Conduct and honor offenses are recorded in the MIDS database. 
This database provides easy access to Academy leadership. 


e. Conclusion 


The performance measurement procedures associated with this objective 
deal specifically with individual performance and not program evaluation. Therefore, the 
performance measurement of indoctrinating and training plebes in military discipline, 
military courtesy, requirements and traits of special trust and confidence, teamwork and 
personal accountability is not effective. Teamwork data are not collected due to the 
complex nature of the subject matter. Moreover, data collection methods for the other 
Ols do not gather information on a class-wide basis in order to afford comparison with 


previous performance or targets. 


Plebe summer personnel could not effectively measure the success of this 


objective even if class-wide data was available. This is due to the complete lack of 
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benchmarks for data comparison. Although individual performance benchmarks are 
identified, they do not assist in assessing objective success. 


3. Provide Basic Training 


The third defined objective of Plebe Summer is to provide plebes with basic 
training in various military subjects including: seamanship, marksmanship, military drill, 
watch standing, and customs and traditions of the Naval Service. This objective is 
designed to develop within plebes a skill set that will enhance their success at the Naval 
Academy and in the fleet as Naval officers. 


a. Outcome Identification 


No outcome for this objective is defined through Academy instruction. 


The Commandant of Midshipmen, however, articulates the intent of this objective: 


Do midshipmen at the Naval Academy need to shoot weapons? The 
answer is no, they don’t need to shoot weapons. But do we produce a 
better midshipman and ultimately a better officer if they’ve had the 
opportunity to go through the process of learning the techniques and 
procedures associated with the surgical application of marksmanship? 
The answer to that question is yes. Not because they will ever have to fire 
that weapon again. But I contend that an officer of the Naval service, 
regardless of warfare specialty he or she serves in, is a more confident 
officer, is an officer who feels better about their own role in this 
profession of arms if they have some sense of their ability to handle 
weapons. 


In conjunction with the martial arts program we’re creating this gives 
them a sense that weapons handling, whether you’re standing on the 
bridge of a ship, staring down a periscope, flying a fighter, or leading a 
rifle platoon, weapons handling is part-in-parcel with the profession of 
arms. 


It’s just good for your self-confidence. So, that’s why we do basic 
marksmanship. Basic drill, of course, is basic teamwork and immediate 
obedience to orders. Basic drill, close-order drill is part-in-parcel with any 
early regimentation and any _ self-imposed discipline. (Personal 
communication, April 2002) 


Plebe Summer experts identified the outcome they associate with this 
objective as: plebes are equipped with the skills to succeed within the academy’s culture. 
Plebes draw on the tools acquired in previous training in order to meet professional 
challenges of watch standing, seamanship, marksmanship, and military drill. Through 
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instruction in the customs and traditions of the Naval service plebes learn the context in 


which they apply these skills. 


Although no outcome is defined through Plebe Summer instruction the 


outcome identified by the Commandant of Midshipman will be used for analysis. Table 


3, Basic Training Worksheet, displays the results of this analysis. 


Table 3. Basic Training Worksheet 


Outcome Outcome Indicators Effective Data 
Collection 
Procedures 


1. Plebes are 1. Percent passing 
Equipped with Skills academic exams 


Reuied ay UeNe 2. Percent Receiving 


sailing qualification 


3. Percent receiving 
weapons qualifications 


4. Percent of Companies 
receiving excellent drill 
scores 


b. Outcome Indicators 


Effective Benchmarks 


1. N Undividual only) 
2. N (Individual only) 
3. N (Individual only) 
4. N (Company only) 


Effective/ 
Non Effective 





Outcome indicators used as metrics for this outcome measure skill level 


and subject understanding. Indicators currently used to measure success of this outcome 


include: 


e Percent receiving passing marks on professional instruction in Damage 


Control, Visual Signaling. 


e Percent earning sailing “B” qualification 


e Percent receiving expert marksmanship scores 


e Percent receiving excellent military drill scores 


The indicators measure only the practical application and understanding of instruction. 


The information obtained through these indicators does not provide feedback in the 


context of naval customs and tradition. 


67 


The indicators measure important aspects of the desired outcome. The 
following paragraphs detail the professional curriculum as well as the curriculum 
objectives described in this objective. Moreover, each subject description will list the 
skills and knowledge measured by these indicators as well as their practical application 


within the Academy’s culture. 


Damage control is a basic skill of all sailors. The ability to fight casualties 
onboard a ship is the foremost challenge of new personnel. Demonstrated proficiency in 
casualty and damage control is a hallmark of professional naval officers. Through 
developing expertise in damage assessment and casualty combat plebes develop and gain 
confidence in the use of personal decision-making skills. The damage control curriculum 
plebes receive includes fire-fighting techniques, flood control, damage control equipment 
and its applications. Plebes also receive hands-on experience plugging and patching 
piping, donning emergency breathing devices, using fire extinguishers, and using the 


Navy all-purpose nozzle and fire hose (USNA, 2001a, p.35). 


Naval ships employ visual communications regularly. Midshipmen are 
expected to attain the ability to encode and decode visual signals rapidly. This 
professional skill can be directly applied to fleet demands and achieving visual 
communications proficiency provides plebes with greater understanding of the demands 
placed on operational naval units. The visual communications curriculum includes 
lessons on the importance and use of visual communications, communications 
organization format, and signaling responsibilities. Plebes receive hands-on training with 


Morse code and Flag hoist signaling (USNA, 2001a, p.37). 


Ship control is the fundamental skill of a naval officer. Regardless of 
warfare specialty, these lessons promote the development of skills such as foresight, 
planning, tenacity, flexibility, and diligence. It matters little whether plebes eventually 
apply these lessons to ships, planes, or riffle platoons; what matters is the development of 
fundamental skills to apply in an environment that is constantly changing to produce 
varying challenges. Proficiency in this subject provides plebes with confidence in their 


ability to work within a constantly changing environment. he sailing curriculum includes 
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lessons in basic seamanship, sailing procedures, sailing equipment, emergencies, and 


sailing practical (USNA, 2001a, p.36). 


The value of marksmanship training was expressed previously. These 
lessons provide plebes with self-confidence and understanding of their roles within a 
military profession. The curriculum includes training in weapon care and safety, firing 
procedures, and live-fire qualification. Midshipmen who score high on marksmanship 
testing qualify for ribbons worn on the uniform. There are three levels of qualification 
for both the M-16 rifle and the 9mm pistol are, expert (E), sharpshooter (S), and basic 
ribbon qualification (USNA, 2001a, p.41). 


Military drill introduces plebes to teamwork and self-discipline. Lessons 
within this subject enlighten plebes to their role within a military environment. 
Moreover, military drill trains midshipmen in the skills of teamwork and obedience to 


military orders. 


The Ols for this objective do not overlap with one another and are 
controllable by the Plebe Summer program. Each subject is tested separately and the 
indicators provide relevant feedback to both Academy leadership and plebes. Successful 
completion of each examination and practical test indicates understanding of the 
curriculum concepts and demonstrates the requisite skills required for later plebe success. 


c. Data Collection Methods 


There are no data collection procedures for this objective. Although there 
are grades associated with the Damage Control, Visual Signals, and Sailing exams, the 
performance of individual plebes is neither recorded nor tracked on a class-wide basis. In 
the area of weapons training, midshipmen who qualify for a ribbon have the qualification 
noted in their military record. However, the rate of qualification among the plebe class is 
not recorded. The same is true for sailing “B” qualification. Each plebe successfully 
qualifying has it noted in their military record. However, the rate of qualification among 


the entire class is not recorded. 


Drill performance is recorded on a company level only and no historical 


data is collected. Therefore, class performance cannot be compared. Also, drill is graded 
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on a company wide basis only. This is problematic in view of the fact individual plebe 
performance cannot be differentiated from the company’s. Poor drill performance on an 
individual basis is not recorded resulting in difficulties identifying and remediating low 
performers. 


d. Benchmarks 


There are no benchmarks defined by Naval Academy instruction that 
allows for comparison among groups or previous classes. Plebe summer personnel could 
not effectively measure the success of this objective even class-wide data was available. 
This is due to the complete lack of benchmarks for data comparison. Although individual 
performance benchmarks are identified, they do not assist in assessing objective success. 


e. Conclusion 


The performance measurement and outcome management procedures for 
this objective are not effective. No procedures for measuring class-wide performance are 
detailed by Academy instruction. This forces academy personnel to develop individual 


performance measures to apply on an individual basis. 


Although the desired outcome and outcome indicators are clearly defined 
by Plebe Summer experts, they are unable to identify clear data collection methods or 
benchmarks that adequately conclude the success or failure in meeting this objective. 
Moreover, data collection methods and benchmarks used by Plebe Summer experts all 
measure individual or company-wide performance and are not amassed to provide class- 
wide comparison with previous performance or pre-selected targets. 


4. Prepare Plebe Class For Integration Into The Brigade of Midshipmen 


The last objective is not defined by Naval Academy instruction. However, it was 
universally identified by Plebe Summer experts and was defined as: preparing the plebe 
class for integration/incorporation into the Brigade of Midshipmen. This objective ties 
the previous three objectives together in that it is focused on teaching the plebe class how 
to apply the previously analyzed tools and skills within the Naval Academy and military 


culture. One expert described this objective: 
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I would say the things that plebes learn to make them good plebes are the 
same things that will make them good midshipmen and good naval 
officers. They are not separate. The aspects of punctuality, being able to 
prioritize, of being able to press on when tired, hungry, and worn out. All 
these qualities we try to impart during plebe summer and see how the do 
with the struggles. Some rise to the challenge, others do not perform as 
well. Nevertheless, those same traits that exude during plebe summer are 
the same that third-class midshipmen, second-class midshipmen, and 
firsties are all tested on and expect in plebes. And they are one in the 
same to those that make a great naval officer (Personal Communication, 
Capt Tyrel Moxey, April 2002). 


Another expert described this objective more directly: 


Under the heightened supervision of Plebe Summer and heightened 
intensity of Plebe Summer we take plebes and turn them into a product 
that we can place into the academic year under less supervision and let the 
brigade finish the training. Because that is where the majority of training 
is going to take place (Personal Communication, Capt Paul Funk, April 
2002) 


The Commandant of Midshipmen underscores the importance of this objective and the 
importance of the transition from Plebe Summer to the academic year: 
Now, the institution here has not done a very good job. Has not done a 
very good job of explaining the seam , the S-E-A-M, between Plebe 
Summer and the formation of the brigade. As well, we have not done a 
very good job as an institution of picking up the momentum of the plebe 
development, the fourthclass development, the professional, moral, 
mental, and physical that occurs during Plebe Summer and have that 


momentum continue in the academic year (Personal Communication, 
Colonel John Allen, April 2002). 


The foundation of this objective is that the professional development of plebes 
does not stop upon completion of Plebe Summer. Rather, it continues throughout the 
fourth-class academic year where the degree of supervision and control place upon plebes 
is greatly reduced. With this in mind, Plebe Summer must inculcate each plebe with 
methods and strategies to effectively apply the tools and skills learned during plebe 


summer in order to be successful during the academic year. 
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a. Outcome Identification and Analysis 


The outcome associated with this objective is: plebe development 
continues during the academic year. This outcome is based on the proper application of 
the tools and skills learned in Plebe Summer. 

I typically do not look at Plebe Summer in isolation. Because, if we’ve 
done it right, Plebe Summer simply prepares the fourth-classmen to join 
the brigade where the preponderance of training continues. Plebe Summer 
simply, through the mechanism of imposed discipline and the stressors 
and strains of a schedule that they can never meet, provides them the 


context and the early military skills and qualities necessary to function 


properly within the brigade (Personal Communication, Colonel John 
Allen, April 2002). 


At first glance, this outcome seemingly duplicates the outcome describe 
for the first objective. This last objective may seem as though it is an extended 
enculturation process. However, while two outcomes ae closely related and rely on 
similar outcome indicators and benchmarks there are distinct differences between the 
two. The first outcome is intended to improve plebe performance as a function of 
successful enculturation during Plebe Summer. Moreover, the first objective measures 
success within the timeline of Plebe Summer. However, this fourth outcome addresses 
continued plebe development upon the conclusion of Plebe Summer as a function of the 
application of skills and tools attained during plebe summer. Also, the outcome 
indicators, data collection, and benchmark comparisons associated with this last objective 
all take place after plebe summer concludes. The results of analysis for this objective is 
presented in Table 4, Transition to Brigade Preparation Worksheet. 


b. Outcome Indicator Analysis 


The Ols listed for this objective measure the professional success of 


midshipmen during the academic year. 
e Percent receiving passing academic grades and overall class GPA 


e = Percent receiving passing academic year military performance grades 


and overall MQPR 


e Attrition rate during the academic year 
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e Incidence of conduct or honor offences during academic year 
e Proportion of class passing PRT during academic year 


These OIs each measure an important aspect of the outcome. Superior 
grades in academics, professional military performance, and PRT indicate plebes 
competence in applying time management and other skills required for success at the 
academy. Moreover, military performance grades measure the day-to-day application of 
the professional training plebes received during Plebe Summer. This OI is particularly 
important because it measures the depth of professional growth during the academic year. 
The rate of honor and conduct offenses measures plebe ability to apply the concepts of 
special trust and confidence as well as personal accountability detailed earlier in this 
research. The rate of attrition measures overall capacity of the plebe class in adjusting to 


and meeting the continued demands after academic year. 


There is little overlap between Ols. Each indicator measures a unique 
aspect of the desired outcome. However, conduct and honor offenses make up a small 
proportion of the military performance grade. Also, conduct and honor offenses can be 


the root cause of attrition. 


Relevant feedback is provided by these OIs. Low academic or military 
performance grades are indicators that plebes are having difficulty meeting the day-to- 
day challenges of the Academy. High rates of conduct or honor offenses indicate plebe 
inability in applying aspects of personal accountability and discipline. Physical 
Readiness Test grades indicate plebe motivation in maintaining the high level of fitness 


acquired during Plebe Summer. 


Although the curriculum is designed to provide plebes with the skills, 
tools, as well as methods to apply them within the Academy’s culture only small portions 
of these OlIs are controllable by the Plebe Summer curriculum. Continued plebe 
development after Plebe Summer is dependent on other outside forces such as the 
motivation and ability of Academy personnel in this continual development. 


Furthermore, individual plebe personal situations vary widely. The effect of staff 
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motivation and training or the plebe personal lives, outside the academy, are not taken 


into consideration with these indicators 


Cc Data Collection Method Analysis 


No data collection procedures are specifically designed to support Plebe 
Summer performance measurement. However, useful data is collected for each indicator 
that allows for comparison among the entire plebe class and between plebe years. 


Indicator data is collected and stored in the MIDS database 


Table 4. Transition to Brigade Preparation Worksheet 


Outcome Outcome Indicators Effective Data Effective Benchmarks Effective/ 
Collection 


Procedures Non Effective 


Plebe Development 1. Academic Grades 
and Performance 
Continues during 
Academic Year 


2. Military Performance 
Grades 


3. Attrition Rate during 
Academic Year 


4. Honor/Conduct 
5. PRT Scores 





d. Benchmark Analysis 


The indicators for this objective have strong benchmarks. They are well 
defined and are easy to collect and apply class-wide. Moreover, the benchmarks provide 
for easy comparison with previous classes or pre-determined targets. As reported earlier, 
conduct and honor offenses have no minimum targets by can be easily applied to 


comparisons with previous classes. 


Grade point average and Military performance grades (MQPR) are 
reported and recorded in the MIDS database. A minimum GPA and MQPR of 2.0 out of 
4.0 is required to pass. Minimum scores for the PRT vary with age and gender. 


However, specific minimum scores are identified. Moreover, PRT scores can be 
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collected class-wide for comparison against pre-selected targets or previous plebe 


performance. 


Attrition rates and incidence of honor/conduct offences are collected in the 
MIDS database. There are no minimum benchmarks for these indicators. The data 
collected is, however, easily compared with previous plebe classes. 


e. Conclusions 


There are no specific performance measurement procedures defined for 
this objective. Nevertheless, the ad hoc performance measurement methods applied to 
this objective by academy personnel are more successful in assessing objective success 


than the previous three objectives. 


The objective and its outcome are very well articulated and contain no 
undefined terms. This allows academy personnel to better identify indicators, data 


sources, and benchmarks. 


Data is collected that is easily accessed and provides for easy comparison 
inter-class as well as intra-class. Furthermore, clear performance benchmarks are 
available for three of five indicators that provide easy comparison with previous 


performance or pre-selected targets. 
D. CHAPTER SUMMARY 


This chapter analyzed the performance measurement and outcome management 
procedures used to assess the success of the Plebe Summer program in meeting its 
objectives. Four objectives were identified followed by an analysis of the performance 


measurement and outcome management procedures applied to each objective. 


Only one of four objectives is supported by effective performance measurement 
and outcome management procedures. Moreover, each objective is supported by 
procedures that are improvised due to the lack of defined performance measurement and 


outcome measurement procedures in Plebe Summer governing instructions. 


The following chapter provides a summary of the complete research, research 


conclusions, and recommendations for the Naval Academy and further research. 
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VI. SUMMARY, CONCLUSIONS, AND REOMENDATIONS 


A. SUMMARY 


The purpose of this research was to assess the effectiveness of the Plebe Summer 
performance measurement and outcome management procedures in measuring the 
success of Plebe Summer objectives. First, the United States Naval Academy and its 
mission were introduced. The Academy has been the premier source of Naval officers 
for over 150 years focusing on the development of young men and women morally, 
mentally, and physically. The role of Plebe Summer as the foundation of midshipman 
development was then presented. Plebe Summer is a diverse program of cultural 
indoctrination and training designed to produce fourthclass midshipmen who function 
effectively in the Academy’s brigade of midshipmen. Plebe Summer is the first military 


training evolution for most members of the incoming plebe class (USNA, 2001b, p. 1-1). 


The principles and concepts of performance measurement and outcome 
management were next introduced as a management tool designed to assist program 
managers dealing with increasingly complex systems. Performance measurement, in 
short, is the process of assessing progress toward organizational goals, including the 
efficiency with which resources are changed into outputs, the quality of outputs, 
outcomes achieved and the overall effectiveness of the organization’s efforts toward their 


mission (NPR, 1997, p. 4). 


Research shows that the concepts of performance measurement can assist 
personnel leading programs such as Plebe Summer. Zigon (1998) reports four basic 
motives to incorporate performance measurement as a part of any management system: 1) 
you cannot manage what you cannot measure, 2) you cannot improve what you cannot 
measure, 3) high performance teams require clear goals, and 4) reward systems require 
metrics. Additional research, directed specifically toward public sector management, 
indicates that management based on performance measurement systems provide greater 
accountability, increased service to the public, and increased public participation (NCPP, 


1997, p. 3). 
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The research began with a thorough literature review of all Naval Academy 
instructions governing Plebe Summer and current literature encompassing performance 
measurement and outcome management. The literature review had two purposes. The 
first purpose was to identify Plebe Summer objectives and performance measurement 
techniques defined by Naval Academy instruction. The second purpose of the literature 
review was to examine the latest research and practical applications of performance 


measurement in public and private sectors. 


Next pre-interview questionnaires and semi-structured interviews of six Plebe 
Summer experts were conducted. The pre-interview questionnaires and interviews were 
designed using Hatry and Kopczynski’s (1997) Guide to Program Outcome 
Measurement. The purpose of the questionnaires and interviews was to collect data from 
Plebe Summer experts concerning the identification and use of performance measurement 
and outcome management procedures as they are practically applied by to measuring the 
success of attaining Plebe Summer objectives. The literature review and interviews 


defined four Plebe Summer objectives: 
e Effect the transition from civilian or enlisted to officer candidate status. 


e Indoctrinate and train midshipmen in proper military discipline, courtesy, 
requirements and traits of special trust and confidence, teamwork, and 


personal accountability. 


e Provide basic training in seamanship, marksmanship, military drill, watch 
standing, and customs and traditions of the Naval service (USNA, 2001b, p. 
6). 


e Prepare the Plebe class for integration/incorporation into the Brigade of 


Midshipmen and the challenge of Plebe academic year. 


Naval Academy performance measurement and outcome management techniques 
applied to each objective were then evaluated using criteria generated by Hatry and 
Kopcezynski’s (1997) Guide to Program Outcome Measurement for the U.S. Department 
of Education. Experts provided information concerning the outcomes, outcome 


indicators, data collection methods, and benchmarks applied to each objective. 
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B. CONCLUSIONS 


The Plebe Summer performance measurement procedures are inadequate. There 
is no specific performance measurement program designed to support Plebe Summer 
evaluation. The terms and definitions associated with Plebe Summer objectives are not 
clear to all program staff. Moreover, only three of four objectives are defined by 
governing Plebe Summer instructions. Although data collection and evaluation occurs, it 
is based on individual performance rather than program-wide evaluation. Additionally, no 
quality benchmarks are defined that supports a qualitative evaluation of success. 
Therefore, ad hoc procedures implemented by Plebe Summer experts are insufficient to 


assess successful, class-wide, objective achievement. 


Terms such as “officer candidate status”, “special trust and confidence’, and 
“teamwork” must be better defined. Lacking definitive terms, Naval Academy staff is 
unable to identify specific outcomes that should be monitored as called for by Hatry and 


Kopcezynski’s (1997). 


The absence of the fourth objective, (preparing the midshipmen for integration 
into the brigade of midshipmen) from Plebe Summer instructions places a great deal of 
stress on Plebe Summer leadership in measuring the objective’s successful achievement. 
Even though the last objective was not defined by Plebe Summer instructions, it is the 
only one of the four identified objectives that is supported by effective performance 
measurement and outcome management procedures. Therefore, it is the only objective 
with which Plebe Summer leaders are able to make a qualitative judgment on whether the 
objective was successfully met or not. Perrin (1998) described how varying 
interpretations of terms and concepts result in measures becoming meaningless in the 
long term. This is an issue for this Plebe Summer objective because it is an improvised 
objective and not defined by instruction. Outcomes, outcome indicators, data collection, 
and benchmarks used for performance measurement of this objective are applied 


unevenly throughout the brigade and in an ad hoc fashion. 


Data is currently collected for a variety of plebe activities such as PRT, academic 


tests, military performance, rate knowledge, and marksmanship. However, in most cases, 
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the data collected is not compiled in a central database to represent class-wide 
performance. This results in the inability to assess program effectiveness. Lacking class- 
wide-data, outcome indicators are irrelevant because they do not support the program 
wide-objectives specified in governing instructions. 
Cc RECOMMENDATIONS 

1. For the Naval Academy 


Plebe Summer is an important component in achieving the Naval Academy’s 
mission. This study suggests that the Naval Academy can apply performance 
measurement and outcome management to assess the quality of Plebe Summer. The 
Naval Academy should establish a working group to oversee the development of an 
outcome measurement process for the Plebe Summer program. The following 
recommendations outline how the Naval Academy can assist a working group in this 


endeavor. 


The first step in developing an effective use of Plebe Summer data is to better 
define Plebe Summer objectives. While three of four objectives are described through 
the governing instructions, the purposes of each objective are not defined. Moreover, 
undefined terms require ad hoc interpretation by staff. The objectives should be modified 
in order to clarify expectations. This will allow a working group to more accurately 
identify the outcomes, outcome indicators, and benchmarks required to support an 


effective measurement process. 


Data collection procedures that support program-wide assessment should be 
designed. Benchmarks are rendered ineffectual without adequate class-wide data that 
supports comparison among and between years. An important aspect of Plebe Summer 
data is not collected. Specifically, no data is collected on the opinions of plebes as the 
recipients of Plebe Summer training. The Naval Academy should develop a method to 
survey plebes on their attitudes and opinions concerning the effects of the curriculum in 
preparing them for the challenges of Plebe Year. Topics such as teamwork, special trust 
and confidence, and discipline can be assessed using such a survey method. 
Additionally, the academy should survey staff and upper-class midshipmen in order to 


assess their opinions of program wide performance and the performance of the plebe 
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class during the academic year. For instance, staff and upper-class midshipmen can 
provide valuable data on what areas of plebe performance require further development. 
This type of data would assist a working group in developing and recommending Plebe 


Summer program improvements. 


Benchmarks must be identified by Naval Academy leadership. No performance 
assessment can occur in the absence of useful comparison structures. While PRT scores 
and Grade Point Averages have minimum allowable scores, the academy has not 
specified any level of quality performance that supports comparison between plebe years. 
The question “is the plebe class performance good or bad?” cannot be answered. 
Furthermore, benchmarks identifying pre-selected targets are not d&fined. A working 
group could recommend possible benchmarks; however the academy leadership must 
define specific levels of performance qualifications. 


2. Recommendations for Further Research 


This research focused on the performance measurement and outcome 
measurement procedures applied to Plebe Summer. There are additional research areas 
that emerged throughout the conduct of this study. The following paragraphs describe 
some additional recommended research. There is no detailed performance measurement 
program specifically designed to support Plebe Summer. Most performance 
measurement and outcome management is conducted using improvised procedures. 
Therefore, the additional recommended research can be conducted with the precept of 


developing such a performance measurement system. 


No study has been conducted that measures the opinions of plebes concerning 
their feelings toward the effectiveness of Plebe Summer in preparing them for the 
challenges of the Naval Academy. This study recommends the development of a survey 
to identify the opinions and attitudes of plebes towards the Plebe Summer curriculum. 
Plebe attitudes towards the Plebe Summer curriculum should be studied before such a 
survey is developed. The purpose of the study would be to answer the questions, “What 
areas of the Plebe Summer curriculum do plebes identify as valuable?” and “Do plebes, 
upper-class midshipmen, and academy staff share common opinions of Plebe Summer 


curriculum value?” 
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The absence of benchmarks was a common theme throughout the research. A 
study could be conducted that would identify levels of performance as they pertain to 
Plebe Summer. A study such as this would identify benchmarks and performance 
qualifiers such as “excellent” or “outstanding” that can be attached to class-wide 
performance. This study could answer the question, “How can plebe performance be 


classified?” 
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APPENDIX A - PLEBE SUMMER TRAINING TOPICS 


I. Command Religious Program 


, . mae Time 
Category Brief Description 
Brief CRP 1 Command Religious Program, Plebe Summer 1 
Indoctrination 
Interpersonal Relations Training (Three 
sessions) 


Brief CRP 2 Command Religious Programs Academic Year 1 
Indoctrination 


Il. Physical Education 


Exam USMC O/C Marine Corps Obstacle Course 
Timed 


Squad Combat Course 
SER (2) Squad Endurance Run 


Tast Crew/Box/ | Introduction to Competitive 
Gym/Wres__| Sports and Close Quarters Combat 


Hand to Hand Combat Instruction (1-3) 





Brief | - __—| Intramural/Intercollegiate Sports 
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PCON Personal Conditioning (1-3) 





2 
if 


}OpEvnt | = __—| Basketball Competition 
|OpEvnt | = | Soccer Competition 
|OpEvnt_ | - | Wrestling Competition 


|OpEvnt | | Boxing Smoker 
|OpEvnt___| = _| All Comers Track Meet 





II. Academic 


[Exam [- | Placement Validation Exams | Vat 
[Brief _—[-__—| Asssociate Dean Academic Affairs 
|Brief__—|- | Academic Dean and Provost Remarks 
[Inst [= | Ac Handbook 
Inst |= | Ac Advisor/Study skills 
|AdmnEvnt_[- | Computer Orientation 2 
|AdmnEvnt_[- | ACE/MBTD 


N 
Nn NTRILN] Re lRlRe 





IV. Supply 


Haircuts (1-3) 
Book Issue 
Repair Tailor Shop Alteration I 
Repair Tailor Shop Alteration II 
Repair Tailor Shop Alteration II 
Contract Tailor Shop 
USNA Store Issue 

d 

d 

d 


USNA Store Exchange 
A 


|AdmnEvnt |- ———_—| Uniform Issue Verification 
FAdmnEvnt_[- | Computer Issue 


eo 
N -Q 
<<) Qa 
Wl ele | np Rel Rel |Re| Re] Go 
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V. Administrative 


PHOTO Photographs for Performance/Academic Records 
jAdmnEvnt |- | Urinalysis, TS 


jAdmnEvnt [ID Military ID Processing/Finger Printing/EPSQ 
Review 


EPSQ EPSQ Brief 





VI. Medical/Dental 


DEN 1 Dental Exam 1 PS al 
DEN 2 Dental Exam 2 | 
Ins 
Bri 


RelN 


FIRSTAID 
Brief _—|- | Health, Hygiene and STD Brief 
[Brief |= ST TRICARE Brief 
Brief —|- | Smoking Cessation Brief 
Brief |= | Drug and Alcohol Brief 
|Brief ==|- ~~ _| Eating Disorders Brief 


f 


Desig Brief Description 


Brief ProDev Department of Professional Development 
Introduction Brief 


Career Brief 

Damage Control 1-3 (P- 100) 
Signaling 1-4 (P-100) 

Sailing Knockabouts 1-4 (P- 100) 
Sailing Lasers 1-3 (P- 100) 
Sailing Navy 44’s (P- 100) 

Study Time 


— 
o>) ee eee NO] 
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VII. Commandant 


[Brief |__| Superintendent’s Call aes 
[Brief | = | Commandant’s Calls [ie 


Brief [|__| 
Sat —— 
[Brief [|| 
[Brief [| _- | 


Sponsor 
D&B 
MA 


OIC Calls 
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Nl 


| VAR | 
| VAR | 
| VAR | 
| VAR | 
| VAR | 
Pe a 
ae 


— 
-) 


AR 
AR 
AR 
AR 
AR 
4 
1 
4 
1 





— 


Squad Leader Instruction Periods (1-22) 22 
Squad Leader Training Periods (1-5) 
Room Inspection Preparation 
Formal Room Inspection 
Personnel Inspection Preparation 
Formal Personnel Inspection 
Sea Bag Inspection 
Phone Calls 

360 FB_ | 360 degree feedback 


Fourth of July Program 

8 & I Sunset Parade 

Plebe Rate Competition/Sing Along 
Detail Set Turnover 


Regimental Picnic/Field Meet 
OpEvnt Holocaust Museum/Vietnam Memorial/Navy 
P Memorial Visit 


Baseball Game 
Formal Regimental Dinner 


Parents’ Weekend 


— 
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APPENDIX B - SQUAD LEADER INSTRUCTION LESSONS 


Lesson Title 


. Plebe Rates/Oath of Office 


. Watch standing 


3. Personal Hygiene 


. Bancroft Hall Orientation 


5. Yard Tour 


16. 


7: 


. Honor I 


. Naval Customs and Traditions 


. Naval Organization 


. Mission of the Naval Academy 


. COC/Followership 


. Loyalty/Communication 


. Good Order and Discipline 


. Leadership Development 


. Enlisted Personnel at USNA 


. Military Performance System 


Admin Conduct System 


Honor II 


Short Description 


An overview of plebe rates along with a detailed 
discussion of the Oath of Office. 


Presentation of Watch standing procedures. 


Basic practices of personal hygiene and proper care 
and prevention of foot blisters and skin rashes. 


Discussion and tour of the Bancroft Hall layout. 


A walking tour of key buildings and monuments in 
the Yard 


Introduction to the Naval Academy Honor Concept. 
An overview of Naval and Marine Corps 
history, heritage, and traditions. 


An overview of the organizational structure of the 
Department of the Navy. 


Review and gain a more in depth understanding of 
the mission of the Naval Academy. 


Review the structure, purpose, and benefits of the 
chain of command and good followership. 


Review concepts of initiative, loyalty, and 
communication. 


A discussion of the importance of good order and 
discipline to a unit. 


A discussion of leadership and how professional 
development is accomplished at USNA 


Review of the role of enlisted personnel at USNA 


A discussion of the Military Performance System, 
MOQPR, and Fitreps. 


A. discussion of the Administrative Conduct 


System, rules, and regulations. 


A discussion of the ethical standards required of 
USNA Midshipmen 
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Lesson Title 


18. 


19. 


20. 


21. 


22. 


Military Etiquette 


Main Office Watch standing 


Leave/Liberty 


Study Skills/Academics 


Joining the Brigade 


Short Description 


Explanation of the basic principles of military 
etiquette and protocol. 


A discussion of the basic requirements to standing a 
Main Office watch. 


A review of Midshipmen leave and liberty policies, 
and Parent’s Weekend regulations. 


A discussion of the importance of academics and 
study method suggestions. 


Acquaint Plebes with knowledge necessary to 
transition to the Academic Year. 
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APPENDIX C —- EXPERT PRE-INTERVIEW WORKSHEET AND 
DEFFINITIONS 


Expert Pre-Interview Worksheet 


The purpose of this interview is to develop an understanding of how Plebe 
Summer success is measured. This interview is being done as part of an analysis of 
performance measures used in Plebe Summer for my Master’s Thesis. This interview 
will attempt to identify the following: 

a. The objectives of Plebe Summer 

b. The outcomes each objective is designed to achieve 
c. Indicators of outcome success 

d. Data sources for outcome indicators 

e. Performance Benchmarks for each outcome indicator 

Although Plebe Summer has dual purposes, my thesis concentrates only on the 
training of incoming plebes. The leadership training of upper class midshipmen is not 
included in this study. 

This interview is not an opinion poll. I want to identify exactly how the Naval 
Academy decides whether Plebe Summer is successful or not (and at what level). This 
pre-interview is designed to provoke thought, not limit scope. If you feel there are items 
(objectives, outcomes, indicators, etc) missing or requiring expansion please add them. 

Key questions to ask yourself as you work through the worksheet are: What 
information is important to me in judging Plebe Summer success? What are the ultimate 
objectives of Plebe Summer? How do I know if an objective is being met? 

The matrix attached to this coversheet lists the objectives of Plebe Summer as 
stated in COMDTMIDNINST 1510.2B. Following each objective is a list of outcomes, 
outcome indicators, data sources and benchmarks. In many instances no information is 


listed. I ask for your expert opinion on filling in the missing data. 
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Each objective should have one or more associated outcomes. In turn, each 


outcome should be aligned with one or more outcome indicators, data sources, and 


benchmarks. The definitions of these terms are attached. 


As you work through the matrix please consider: 


a. 


In your opinion, does this list of objectives cover all the objectives of 
Plebe Summer? 

In your opinion, what are the outcomes these objectives intended to 
achieve? 

What are the indicators of each outcome? 

How is data on each indicator collected? 

What is the benchmark for each indicator? What are the minimum 


acceptable levels of performance associated with each indicator? 
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DEFINITIONS 


Outcome: Outcomes are not what the program itself did. Rather, outcomes are the 
consequences of what the program did. They provide information on events, 
occurrences, conditions, or changes in attitudes and behavior that indicate progress 
toward achievement of the mission and objectives of the program. Outcomes happen 
outside the program. 


There are two types of outcomes Intermediate Outcomes and End Outcomes. 


Intermediate outcomes are expected to lead to an ends desired, but are ends in 
themselves. Examples are participation in program, improvements in learning practices 


relating to instruction, and increase in hours of study. 


End outcomes are the desired results of the program. These can be either short-term or 


long-term. Increased test scores, reduction of conduct offenses, reduced attrition rates. 


Objectives: This is often attached to the word Mission. It is a statement that tells where 


the program wants to go. What are the overarching results that are planned? 


Outcome Indicators: This is not the same as Outcomes. Outcome indicators specify 
what is to be measured. Specific indicators will depend on the type of data collection 
used. Outcome indicators identify specific numerical values that indicate progress 
toward achieving an outcome (such as percentage or ratio). Examples include percentage 


of persons qualifying expert rifleman. 


Benchmark: What do you compare outcome indicator information to? Benchmarks 
provide a comparison for data. Benchmarks provide the minimum level of performance 
or change acceptable. Major types of benchmarks include: 


Previous Performance (Improvement over time) 
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Performance of Similar Group (Comparison against the best) 
Outcomes of different groups (comparison against the best) 
Pre-selected targets (pre-selected standard) 


Different delivery practices (use of comparison groups) 
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